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ABSTRACT 

This  research  employed  a  case  study  strategy  to  examine  the  benefits  of  using  synthetic  environments  to 
provide  mission  preparation  for  command  and  control  teams  prior  to  a  large-scale,  live  warfighting 
exercise.  During  Pitch  Black  08,  we  compared  the  performance  of  two  Air  Battle  Management  (ABM)  teams 
from  41  Wing,  Surveillance  and  Response  Group:  one  team  prepared  by  supporting  normal  flying 
operations,  the  other  took  part  in  a  virtual  mission  preparation  exercise  named  Black  Skies  08.  The  synthetic 
environment  provided  in  Black  Skies  08  was  designed  to  simulate  Pitch  Black  08  as  closely  as  possible  in 
terms  of  mission  scenarios,  order  of  battle,  airspace,  procedures,  and  tactics.  The  evaluation  of  the  training 
provided  in  the  virtual  mission  preparation  was  carried  out  using  the  four-level  model  developed  by 
Kirkpatrick  (1959):  Reaction,  Learning,  Behavioural,  and  Results  critera.  The  members  of  the  ABM  team 
that  participated  in  Black  Skies  08  reported  that  the  virtual  mission  preparation  compared  favourably  to 
both  regular  training  and  live  exercises  in  terms  of  the  capacity  to  provide  training  experiences  critical  to 
the  ABM  role.  Over  the  course  of  the  virtual  mission  preparation,  the  ABM  team  displayed  a  marked 
improvement  in  teamwork  processes,  motivational  state,  and  across  most  mission  essential  tasks.  During 
the  subsequent  live  warfighting  exercise,  the  teamwork  processes  and  taskwork  performance  of  this  team 
were  rated  as  better  overall  than  those  of  the  matched  ABM  team  that  did  not  take  part  in  virtual  mission 
preparation.  The  virtual  mission  preparation  also  demonstrated  a  broader  organisational  and  operational 
value  to  the  RAAF:  it  provided  the  White  Force  with  the  opportunity  to  identify  deficiencies  in,  and 
mitigate  risks  associated  with,  the  Pitch  Black  mission  scenarios;  and  it  allowed  the  ABM  team  to  refine 
their  Pitch  Black  plans  and  procedures,  enabling  them  to  more  effectively  utilise  the  valuable  training 
opportunity  presented  by  a  large-scale,  live  warfighting  exercise. 
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Exercise  Black  Skies  2008:  Enhancing  Live  Training 
Through  Virtual  Preparation 

Part  One:  An  Evaluation  of  Training  Effectiveness 


Executive  Summary 

The  first  principle  of  Australian  Defence  training  doctrine  is  that  our  forces  should  train  as 
they  operate  (Australian  Defence  Force  Warfare  Centre,  2006).  For  air  warfare  command 
and  control  teams,  however,  the  opportunity  to  take  part  in  training  that  faithfully 
replicates  operational  environments  is  rare,  largely  restricted  to  participation  in  exercises 
such  as  Pitch  Black  and  Talisman  Sabre.  The  scarcity  of  these  valuable  training 
opportunities  has  generated  considerable  interest  in  the  use  of  synthetic  environments  as  a 
means  of  supplementing  or  augmenting  live  training.  InTheFutureA  ir  and  S  pace  0  per  ating 
Concept  (Air  Power  Development  Centre,  2007),  virtual  and  simulated  environments  are 
identified  as  crucial  to  building  appropriate  skill  sets  and  experience  in  Royal  Australian 
Air  Force  (RAAF)  warfighters. 

The  study  described  in  this  report  forms  part  of  a  collaborative  research  program  between 
DSTO's  Air  Operations  Division  and  the  US  AFRL's  Human  Effectiveness  Directorate, 
under  the  auspices  of  Project  Arrangement  P  A  -A  F -0025:  Distributed  M  issi  on  Training 
R  esearch.  This  study  had  both  broad  and  specific  research  goals.  The  broad  research  goal 
was  to  investigate  the  effectiveness  of  a  range  of  tools  and  methods  for  structuring  and 
managing  RAAF  synthetic  team  and  collective  training.  The  method  and  outcomes 
relating  to  this  broad  goal  are  presented  in  a  companion  report  (Tracey,  Hasenbosch, 
Vince,  Pope,  Stott,  Best,  Shanahan,  &  Finch,  2009).  The  specific  research  goal,  and  the  focus 
of  this  report,  was  to  investigate  the  potential  benefits  to  the  RAAF  of  using  synthetic 
environments  to  provide  virtual  mission  preparation  for  live,  large  scale  training  exercises. 

A  case  study  strategy  was  employed  to  examine  the  benefits  of  using  synthetic 
environments  to  provide  mission  preparation  for  a  41WG  Air  Battle  Management  (ABM) 
team  prior  to  Exercise  Pitch  Black  08.  One  ABM  team  -  the  control  team  -  prepared  for 
Pitch  Black  08  by  supporting  normal  flying  operations.  A  second  ABM  team  prepared  by 
taking  part  in  Exercise  Black  Skies  08;  the  synthetic  environment  provided  in  Black  Skies 
08  was  designed  to  simulate  Pitch  Black  08  as  closely  as  possible  in  terms  of  mission 
scenarios,  order  of  battle,  airspace,  procedures,  and  tactics. 

The  evaluation  of  the  training  provided  in  Black  Skies  08  was  carried  out  using  the  four 
levels  of  criteria  suggested  by  Kirkpatrick  (1959):  Information  was  gathered  about  the 
ABM  team's  reactions  to  the  virtual  mission  preparation,  the  learning  that  occurred  over 
the  course  of  the  virtual  mission  preparation,  the  impact  of  the  virtual  mission  preparation 
on  performance  in  the  subsequent  live  warfighting  exercise,  and  the  broader  value  to  the 
organisation  of  the  virtual  mission  preparation. 


The  outcomes  from  this  study  provide  support  for  the  view  that  synthetic  training 
technologies  can  have  a  significant  impact  on  the  ability  of  RAAF  warfighting  teams  to 
perform  their  mission  in  large-scale,  complex,  and  dynamic  warfighting  situations.  The 
ABM  team  that  took  part  in  Black  Skies  08  provided  positive  evaluations  of  the  training 
experience,  their  performance  improved  over  the  course  of  the  virtual  mission 
preparation,  and  during  Pitch  Black  08  they  outperformed  the  ABM  team  that  prepared 
for  Pitch  Black  08  by  supporting  normal  flying  operations.  In  addition,  the  virtual  mission 
preparation  demonstrated  a  broader  organisational  and  operational  value  to  the  RAAF:  it 
provided  the  White  Force  with  the  opportunity  to  identify  deficiencies  in,  and  mitigate 
risks  associated  with,  the  Pitch  Black  08  mission  scenarios;  it  also  allowed  the  ABM  Team 
to  refine  their  Pitch  Black  08  plans  and  procedures,  enabling  them  to  more  effectively 
utilise  the  valuable  training  opportunity  presented  by  a  large-scale,  live  warfighting 
exercise. 

These  outcomes  lend  weight  to  the  assertion  that  the  RAAF  could  benefit  significantly 
from  the  broad  application  of  synthetic  team  and  collective  training  technologies  (Air 
Power  Development  Centre,  2007);  however,  further  research  is  required  into  factors  that 
influence  the  effectiveness  and  efficiency  of  team  and  collective  training  in  synthetic 
environments.  The  potential  impact  of  these  technologies  will  also  be  constrained  by  their 
method  of  employment  and  by  their  accessibility:  If  experiences  of  the  kind  provided 
during  Black  Skies  are  made  available  infrequently,  and  only  to  small  groups  of 
individuals,  their  full  potential  will  not  be  realised.  This  has  implications  for  the 
simulation  infrastructure  used  to  provide  these  experiences.  It  is  unlikely  that  existing 
RAAF  simulation  assets  will  be  able  to  provide  frequent  and  regular  synthetic  team  and 
collective  training  for  large  numbers  of  operators.  Development  of  the  appropriate 
infrastructure  will  require  significant  investment,  and  it  is  important  that  this  investment 
is  justified  and  guided  by  rigorous  science. 

Key  Recommendations: 

1.  RAAF  should  expand  its  capacity  to  provide  synthetic  team  and  collective  training 
for  its  warfighters. 

2.  RAAF  should  support  further  research  into  the  effective  and  efficient  use  of  synthetic 
team  and  collective  training  technology. 

3.  RAAF  (and  the  ADF  more  broadly)  should  invest  in  dedicated  infrastructure  for 
synthetic  team,  collective,  and  joint  training. 

4.  RAAF  should  utilise  DSTO's  research  experience  to  guide  further  investment  in,  and 
application  of,  tools  and  methods  for  synthetic  team  and  collective  training. 

5.  RAAF  should  expand  its  participation  in  coalition  synthetic  training  exercises,  and 
facilitate  the  conduct  of  collaborative  research  and  development  with  coalition 
partners. 
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1.  Introduction 

Thefirstprincipleof  Australian  Defence  Forcetraining  doctrine  is  that  our  forces  should  train 
as  they  operate  (A  ustral  i  an  Defence  Force  Warfare  Centre,  2006).  For  ai  r  warfare  command 
and  control  teams,  however,  the  opportunity  to  take  part  in  training  that  faithfully  replicates 
or  simulates  operational  circumstances  and  environments  is  rare,  largely  restricted  to 
participation  in  large-scale,  live  exercises  such  as  Pitch  Black  and  Talisman  Sabre.  Thescarcity 
of  these  valuable  training  opportunities  has  generated  considerable  interest  in  the  use  of 
synthetic  environments  as  a  means  of  supplementing  or  augmenting  live  training.  In  The 
Future  Air  and  Space  Operating  Concept  (Air  Power  Development  Centre,  2007),  virtual  and 
simulated  environments  are  identified  as  crucial  to  building  appropriate  skill  sets  and 
experienced  Royal  Australian  Air  Force(RAAF)  warfighters. 

There  are  cl  ear  benefi  ts  to  provi  d  i  ng  team  and  col  I  ecti  ve  trai  ni  ng  i  n  a  sy  ntheti  c  envi  ronment 
(Knerr,  Breaux,  Goldberg,  &  Thurman,  2002;  Schiflett,  Elliott,  Dalrymple,Tessier,&  Cardenas, 
2000).  First,  there  are  significant  savings  associated  with  the  reduced  demand  on  platforms 
and  personnel .  Thi  s  i  s  parti  cul  arl  y  perti  nent  i  n  the  case  of  the  RA  A  F,  w  hi  ch  i  s  constrai  ned  i  n 
its  capacity  to  develop  its  forces  by  vi  rtueof  its  rel  ati  vely  smal  I  size  and  resource  avai  labi  I  ity 
(Air  Power  Development  Centre,  2007).  Second,  it  is  possibleto  generate  complex,  dynamic 
training  scenarios  free  from  safety  or  environmental  concerns.  The  use  of  synthetic 
environments  can  provi  detraining  opportunities  that  cannot  be  made  available  in  the  real 
world,  and  which  in  some  respects  are  more  operationally  valid  than  live  training  (Knerr  et 
al„  2002).  Third,  the  use  of  synthetic  environments  can  enhance  preparedness  by  giving 
command  and  control  teams  the  opportunity  to  train,  as  they  fight,  on  a  more  regular  basis 
(Schiflett  et  al .,  2000). 

W  hat  i  s  not  cl  ear  i  s  the  extent  to  w  h  i  ch  team  and  col  I  ecti  ve  trai  n  i  ng  i  n  a  sy  ntheti  c  envi  ron  ment 
actual  I  y  I  eads  to  more  effecti  ve  performance  i  n  a  I  i  ve  warfi  ghti  ng  envi  ronment.  The  pri  nci  pal 
aim  of  this  study,  in  which  we  provided  a  RAAF  command  and  control  team  with  virtual 
mission  preparation  prior  to  a  live  warfighting  exercise,  was  to  address  this  question.  In  the 
i  ntrod  ucti  on  to  thi  s  report,  f  i  rst  we  cri  ti  cal  I  y  revi  ew  the  I  i  teratu  re  on  the  effecti  veness  of  team 
and  collective  training  in  synthetic  environments.  Second,  we  outline  the  broad  and  specific 
research  goals  of  this  study,  and  state  the  relevant  research  hypotheses. 

1.1  Previous  Research  on  the  Effectiveness  of  T earn  and  Collective 
Training  in  Synthetic  Environments 

The  most  widely  used  framework  for  the  evaluation  of  training  programs  is  Kirkpatrick's 
(1959)  four-level  model  (Goldstein  &  Ford,  2002;  Kraiger,  2002),  comprising  the  criteria 
Reaction,  Learning,  Behavioural,  and  R  esuits.  Reacti  ons  criteria  rel  ateto  whattrainees  thought  of 
the  program.  Learning  criteria  assess  the  extent  to  which  trainees  learnt  the  principles,  facts, 
or  techniques  specified  as  training  objectives.  Behavioural  criteria  are  concerned  with  the 
changes  i  n  job  performance  resul  ti  ng  from  the  trai  ni  ng.  Results  criteri  a  refer  to  the  ulti  mate 
val  ue  of  the  trai  n  i  ng  p  rogram  to  the  organ  i  sati  on ,  such  as  i  mp  rovement  i  n  qu  anti  ty  or  qu  al  i  ty 
of  output,  or  mitigation  of  risk.  The  review  of  the  literature  that  bears  on  the  question  of  the 
effectiveness  of  collective  training  in  synthetic  environments  is  organised  within  this 
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framework;  consistent  with  training  research  more  generally,  however,  thereis  no  literature 
that  explicitly  attempts  to  evaluate  Kirkpatrick's  Results  criteria. 

Thi  s  I  iterature  revi ew  i  s  restri  cted  to  research  i  n  the  ai  r  warfare  domai  n  (for  an  overvi  ew  of 
the  application  and  efficacy  of  virtual  training  in  other  military  domai  ns  see  Stone,  2002,  and 
K  nerr  et  al . ,  2002) .  A I  most  exd  u si  vel  y ,  thi  s  research  i  s  the  prod  uct  of  col  I  aborati  ve  research  by 
the  United  Kingdom's  (UK)  Defence  Science  and  Technology  Laboratory  (DSTL)  and  the 
United  States  (US)  Air  Force  Research  Laboratory  (AFRL).  Over  the  past  decade,  DSTL  and 
AFRL  have  been  involved  in  a  program  of  applied  research  investi  gating  the  use  of  synthetic 
environments  to  provide  collective  training  for  ai  r  operati  ons.  A  number  of  exercises  have 
been  carried  out  in  this  program  of  research,  including  Trial  VirtEgo  (McIntyre,  Smith,  & 
Bennett,  2002),  Trial  SyCLON  E  (Smith  &  M  dntyre,  2003),  and  Exercise  Red  Skies  (Smith  etal., 
2006).  Broadl  y  speaki  ng,  these exerci  ses  have  i  nvol  ved  the creati  on  of  syntheti  c  envi  ronments 
in  which  geographically  dispersed  aircrew  from  a  variety  of  different  roles  -  air-to-air,  air-to- 
ground,  and  to  a  lesser  extent  airborne  warning  and  control  -  participate  in  complex  mission 
scenarios.  These  mission  scenarios  are  based  on  real-world  operational  theatres,  and  are 
designed  to  support  the  aircrew's  training  needs  throughout  the  complete  mission  cycle. 

1.1.1  Evaluations  of  Reactions  Criteria 

A I  though  parti  ci  pant  reach  ons  are  not  the  most  val  i  d  measure  of  trai  ni  ng  effecti  veness,  they 
can  be  used  to  establish  the  degree  of  user  acceptance  in  a  training  system;  favourable 
reactions  to  a  training  program  areoften  a  necessary  condition  for  continued  system  use  and 
development  (Goldstein  &  Ford,  2002;  Schreiber,  Rowe,  &  Bennett,  2006).  Over  a  period  of 
several  years,  from  2002,  the  US  AFRL  (Schreiber  et  al.,  2006)  collected  information  of  this 
nature  from  warfighters  parti  ci  pati  ng  i  n  five-day  trai  ni  ng  research  exercises  atthedistri  buted 
mission  operations  research  site  at  M  esa,  Arizona  ( distributed  mission  operati  ons  is  the  US  term 
for  team  and  collectivetraining  in  synthetic  envi  ronments).  Feedback  about  thegeneral  value 
of  the  distributed  mission  operations  training  was  provided  by  327  F-16  pilots  and  49 
Airborne  Warning  and  Control  Station  (AWACS)  controllers,  and  32  F-16  pilots  provided 
information  about  the  extent  to  which  thetrai  ni  ng  was  capableof  providing  mission  essential 
exp er i  en ces.  O  ver al  I ,  th ere  w  as  a  stron g  u  ser  acceptan ce  of  th e  val  u  e  of  th e  trai  n  i  n g  p rov i  d  ed 
in  distributed  mission  operations.  The  vast  majority  of  participants  agreed  that  the  training 
had  improved  their  combat  mission  readiness,  and  that  they  would  recommend  thetraining 
experi  enceto  other  pi  I  ots  or  control  I  ers.  The di  stri  buted  mi  ssi  on  operati  ons  envi  ronment  was 
also  rated  higher  than  all  other  training  environments  (including  live  exerci  ses  and  regular 
training)  in  its  capacity  to  provide  experiences  critical  to  the  F-16  air-to-air  operational  role. 

Favourable  reacti  ons  to  thecollective  training  provided  in  synthetic  envi  ronments  were  also 
reported  i  n  Tri  al  Vi  rtEgo  (M  cl  ntyre  et  al .,  2002)  and  Exerci  se  Red  Ski  es  (Smith  et  al .,  2006).  The 
aircrew  who  participated  in  Trial  VirtEgo  were  of  the  opinion  that  60%  of  their  role-specific 
mission  essential  tasks  could  be  trained  just  as  well  in  the  synthetic  environment  as  in  a 
regular  I  ivetraining  exercise;  for  28%  of  these  tasks,  the  synthetic  envi  ronment  was  rated  as  a 
better  training  environment  than  live  training  exercises  (McIntyre  et  al.,  2002).  Flowever, 
reacti  ons  to  the  geograp  hi  cal  I  y  d  i  spersed  col  I  ecti  ve  trai  n  i  ng  p  rovi  d  ed  i  n  Tri  al  SyC  L  O  N  E  w  ere 
I  ess  favourable,  particularly  in  relation  to  the  effecti  veness  of  distributed  planning,  briefing, 
and  debriefing  systems  (Smith  &  McIntyre,  2003). 
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1.1.2  Evaluations  of  Learning  Criteria 

There  i  s  al  so  confl  i  cti  ng  evi  dence  regard  i  ng  the  extent  to  w  hi  ch  col  I  ecti  vetrai  ni  ng  i  n  syntheti  c 
envi  ronments  faci  I  itates  the  I  earni  ng  of  teamwork  processes  and  mi  ssi  on  essenti  al  ski  1 1  s.  The 
goal  of  RO  A  DRUNNER98  (Schif  I  ett  et  al . ,  2000)  was  to  i  nvesti  gate  the  i  impact  that  col  I  ecti  ve 
training  in  a  synthetic  environment  has  on  mission  performance.  The  exercisewasuniquein 
two  respects.  First,  it  focused  on  the  performance  of  command  and  control  teams  - 
specifi  cal  I  y,  the  weapons  d  i  rectors  and  ai  r  su  rvei  1 1  ancetechni  ci  ans  that  make  up  an  A  i  rborne 
Warni  ng  and  Control  Systems  (A  WACS)  crew  -  with  fi ghter  ai  rcraft  performi  ng  a  supporti  ng 
role.  Second,  itfocused  on  teamwork  processes  rather  than  mission  essential  tasks.  Based  on 
cognitive  task  analyses  of  the  AWACS  operational  role,  Schiflett  et  al.  (2000)  developed  a 
taxonomy  of  command  and  control  team  performancedesigned  to  capturethe corefunctions 
that  compriseAWACSteamwork  process.  Theprincipal  dimensions  were  mi  ssi  on  planning 
(the  establ  i  shment  of  rol  es,  responsi  bi  I  i  ti  es,  and  conti  ngency  pi  ans),  communi  cati  on  d  i  sci  pi  i  ne 
(the  extent  to  which  individuals  follow  guidelines  for  communication  exchange), 
communication  content  and  timing  (the  extent  to  which  the  right  information  is  exchanged 
with  the  right  person,  at  the  right  time),  and  dynamic  replanning  (the  ability  to  recognise 
when  pi  ans  are  no  longer  effective,  to  generate  alternatives,  and  to  decide  upon  a  new  course 
of  action). 

Over  seven  days,  three  A  WACS  teams  each  took  part  in  seven  complex  missions  designed  to 
pi  ace  the  i  nterdependenci  es  withi  n  theteam  under  stress.  Supporti  ng  rol  es  and  enemy  forces 
wereplayed  either  by  virtual  or  constructiveentities.Theoverall  performanceoftheAWACS 
teams  improved  over  the  seven  mission  scenarios.  Although  the  synthetic  training  did  not 
have  a  differential  impact  on  the  distinct  teamwork  dimensions,  it  did  have  a  differential 
i  mpact  on  the  different  mi  ssi  on  phases,  w  ith  overal  I  performance  i  improvement  attri  buted  to 
better  performance  in  the  mi  ssi  on  execution  and  debriefing  phases  of  each  scenario.  Schiflett 
et  al .  (2000)  reported  that  the  debri  ef  i  ng  experi  ence  was  found  to  be  parti  cu  I  arl  y  benefi  ci  al  to 
the  AWACS  crew:  as  the  duration  of  an  AWACS  flight  extends  well  beyond  the  flight 
duration  of  a  particular  set  of  combat  aircraft,  the  AWACS  crew  do  not  usually  debrief  with 
pilots  after  an  operational  mission. 

The  results  of  Trial  SyCLONE  (Smith  &  McIntyre,  2003)  were  less  positive.  The  goal  of  this 
exercise  was  to  explore  the  relative  benefitto  Royal  Air  Force(RAF)  aircrew  of  distributed 
versus  col  I  ocated  mi  ssi  on  trai  ni  ng.  RA  F  and  U  S  ai  rcrew,  performi  ng  both  ai  r-to-ai  r  and  ai  r-to- 
ground  roles,  took  part  in  two  days  of  virtual  missions.  Not  only  did  the  performance  of  the 
aircrew  fail  to  improve  over  these  missions,  buttherewasinfactadrop  in  performance  across 
a  majority  of  the  mission  objectives.  The  criteria  on  which  the  performance  loss  was  most 
dramatic  included  the  effectiveness  of  mission  planning,  the  degree  to  which  each  team 
member  understood  their  place  in  the  broader  air  operation,  review  of  tactics  as  a  result  of 
lessons  learned,  and  performance  of  role-specific  tasks.  Furthermore,  the  trust  between 
coalition  aircrew  decreased  over  the  course  of  themission.  Smith  and  M  dntyreconduded  that 
geographic  dispersion  during  collective  training  in  synthetic  environments  has  a  negative 
impact  on  training  value,  with  the  UK  and  US  aircrews  unableto  build  rapport  or  develop 
inter-team  cohesion. 
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Smith  and  McIntyre  (2003)  noted  that  on  one  level,  Trial  SyCLON  E  could  be  considered  a 
success:  the  US  and  UK  aircrew  were  able  to  plan,  brief,  fly,  and  debrief  a  complex  mission. 
However,  they  also  pointed  outthatTrial  SyCLONE  wasdearly  not  effective  in  providing 
high  value  mission  training,  and  cautioned  that  simply  linking  simulators  to  provide  a 
common  sy  ntheti  c  battl  espace  i  s  not  suffi  ci  ent  to  provi  de  effecti  vetrai  ni  ng  for  ai  r  operati  ons. 

1.1.3  Evaluations  of  Behavioural  Criteria 

The  extent  to  which  learning  that  occurred  in  a  synthetic  training  environment  transfers  to 
performance  i n  a  I i  ve  warfi ghti ng  envi ronment  was  exami ned  i  n  Tri al  Vi rtEgo  (M  cl  ntyre  et 
al .,  2002)  and  Exercise  Red  Skies  (Smith  et  al .,  2006). 

The  goal  of  Tri  al  V  i  rtEgo  was  to  provi  de  Royal  A  i  r  Force  ( RA  F)  ai  rcrew  w  i  th  vi  rtual  mi  ssi  on 
preparati  on  for  thel  iveflying  phaseof  a  weapons  instructors  course.  Thetrial  was  specifically 
targeted  at  provi  di  ng  trai  ni  ng  i  n  the  hi  gh-level  teamwork  processes  essential  to  successful  ai  r 
operations;  the  criteria  measured  included  communication  and  coordination,  the  ability  to 
build  and  maintain  situation  awareness,  and  to  make  tactical  decisions  in  a  complex  and 
dynamic  environment. 

Inthevirtual  preparation  phase,  the  aircrew  undertaking  theweaponsinstructorscoursetook 
part  in  two  full  missions.  Over  the  two  mission  days,  there  was  an  improvement  in  the 
collective  performance  of  the  aircrew  across  the  majority  of  criteria;  in  most  cases  this 
improvement  was  maintained  in  the  first  mission  of  the  I  iveflying  phase  of  the  weapons 
instructors  course.  However,  ratings  of  performance  on  criteria  related  to  decision  making 
and  adaptability  inthefirst  mission  of  the  I  iveflying  phase  were  substantially  lower  than  in 
the  final  mission  of  the  virtual  mission  preparation.  It  is  also  worth  noting  that  the  virtual 
mission  preparation  had  no  impact  on  thestandard  with  which  aircrew  utilised  role-specific 
skills  (i.e.,  non-collective  taskwork  skills)  within  the  collective  environment;  this  finding  is 
consistent  with  the  results  of  other  synthetic  training  exercises  such  asT  rial  SyCOE  (M  cl  ntyre 
&  Smith,  2000)  and  Trial  SyCLON  E  (Smith  &  McIntyre,  2003). 

Mcl ntyreetal.  (2002)  argued  that thevalueofthetraining  provided  intrial  VirtEgowould  be 
confirmed  if  performance  of  the  candidates  in  the  initial  stages  of  the  I  iveflying  phasewas 
enhanced.  Although  they  reported  that thei nstructors  on  thecourse noted  that thecandidates 
performed  better  than  isusual  onthefirstdayoftheliveflyingphase,thedesignofthestudy 
-  specifically,  the  lack  of  a  control  group  -  did  not  permit  a  rigorous  test  of  this  hypothesis. 
The  particular  nature  of  the  transfer  context  -  a  weapons  instructors  course  -  also  detracts 
from  the  ability  to  generalise  the  findings  of  this  study  to  I  arger-scale  warfighting  contexts. 

Exercise  Red  Skies  (Smith  et  al.,  2006)  was  far  broader  in  scope:  the  primary  goal  was  to 
investigatethe  benefits  of  conducting  virtual  mission  preparation  prior  to  a  large-scale  live 
exercise  (Exercise  Red  Flag).  In  the  synthetic  environment  provided  in  Red  Skies,  RAF  and 
U  SA  F  ai  rcrew  fl  ew  mi  ssi  ons  si  mi  I  ar  to  those  that  they  wou  I  d  I  ater  fl  y  i  n  the  I  i  ve  envi  ronment 
of  Red  Flag. The  mi  ssi  ons  flown  during  the  virtual  and  live  exercises  were  matched  in  terms 
of  coalition  package,  mission  type  (air  interdiction,  close  air  support,  and  time-sensitive 
targeting),  Air  Tasking  Order  (ATO),  rules  of  engagement,  and  geographical  location  (the 
Nellis  Air  Weapons  Range  in  Nevada,  USA).  According  to  Smith  etal.  (2006)  the  only  major 
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difference  between  the  Red  Skies  and  Red  Flag  missions  was  that  (1)  there  were  a  larger 
number  of  aircraft  involved  in  Red  Flag,  and  (2)  both  day  and  night  missions  were  flown 
during  Red  Flag,  whereas  only  day  missions  were  flown  during  Red  Skies. 

I  n  order  to  quantify  thetrai  ni  ng  benefits  of  the  vi  rtual  mi  ssi  on  preparati  on,  Smith  et  al .  (2006) 
used  surveys  to  measure  participants  expectations  about  collective  training  in  synthetic 
environments,  as  well  as  their  reactions  to  the  virtual  mission  preparation  provided  in  Red 
Skies.  The  mission  performance  of  the  aircrew  during  both  the  synthetic  exercise  and  the 
subsequent  live-fly  exercise  was  assessed  by  subject  matter  experts.  Although  Smith  etal.  do 
not  provi  desuffi  ci  ent  i  nformati  on  for  a  meani  ngful  i  nterpretati  on  of  the  survey  results,  post¬ 
exercise  interviews  with  the  aircrew  suggest  that  Red  Skies  was  regarded  as  a  valuable  spin- 
up  activity  for  Red  Flag.  Furthermore,  mission  performance  in  the  live  exercise  was  rated  by 
the  subject  matter  experts  as  appreciably  higher  than  in  the  synthetic  exercise. 

The  Red  Ski  es  study  provi  des  some  evidence  for  the  benefits  of  usi  ng  synthetic  envi  ronments 
in  the  preparation  of  teams  of  warfighters  for  large-scale,  live  training  exercises.  Flowever, 
from  the  data  that  is  reported  it  is  not  possible  to  determine  the  extent  to  which  learning 
occurred  over  the  course  of  thevirtual  mission  preparation.  As  was  the  case  in  Trial  VirtEgo, 
thereisalso  no  evidenceto  indicate  whether  aircrew  that  received  virtual  mission  preparation 
for  Red  Flag  outperformed  aircrew  that  prepared  for  Red  Flag  using  traditional  methods. 

1.2  The  C urrent  Program  of  Research 

The  present  study  forms  part  of  a  collaborative  research  program  between  DSTO's  Air 
Operations  Division  andtheUSAFRL'sFluman  Effectiveness  Directorate,  under  theauspices 
of  Project  Arrangement  PA-AF-0025:  Distributed  M  i  ssi  on  Training  Research.  The  broad  goal 
of  this  program  of  research  is  to  develop  and  evaluate  tools  and  methods  for  providing 
effective,  operational  I  y-relevant,  synthetic  team  and  collective  training  for  air  combat. 
Previous  studies  conducted  under  this  program  include  Pacific  Link  I  (Craneetal.,  2006)  and 
Pacific  Link  II  (Best  etal.,  2007).  Thecurrent  study  had  both  broad  and  specific  research  goals. 
The  broad  research  goal  was  to  investigate  the  effectiveness  of  a  range  of  tool  sand  methods 
for  structuring  and  managing  RAAF  syntheticteam  and  collectivetraining.Themethod  and 
outcomes  relating  to  this  broad  goal  are  presented  in  a  companion  report  (Tracey  etal.,  2009). 
Thespecific  research  goal ,  and  thefocusof  this  report,  was  to  investigate  thepotential  benefits 
to  the  RAAF  of  using  synthetic  envi  ronments  to  provi  demission  preparation  for  live,  large- 
scaletraining  exercises.  The  synthetic  exercise  that  was  used  to  provide  this  virtual  mission 
preparation  was  named  Exercise  Black  Skies  08,  and  was  conducted  in  the  Air  Operations 
Experimentation  Centre  at  DSTO  Mel  bourne  in  May  2008,  two  weeks  prior  to  Pitch  Black  08. 
Pitch  Black  is  a  biennial,  live  air  combat  training  exercise  hosted  by  the  RAAF,  involving 
participants  from  a  number  of  foreign  military  forces  (Department  of  Defence,  2009). 

This  study  builds  upon  the  existing  research  in  a  number  of  ways.  First,  the  study  examines 
the  benefits  of  virtual  mission  preparation  in  theAustralian  warfighting  context,  rather  than 
that  of  the  U  S  or  the  U  K .  The  RA  A  F  i  s  i  ndeed  requ  i  red  to  be  abl  e  to  i  ntegrate  -  to  essenti  al  I  y 
work  as  a  single  system  -  with  the  air  forces  of  the  US  and  the  UK.  Flowever,  as  a  result  of 
Australia's  broader  strategic  context  there  are  significant  cultural,  organisational,  and 
operational  differences  between  the  RAAF  and  the  air  forces  of  its  primary  allies  (Air  Power 
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Development  Centre,  2007).  Second,  weextend  theresearch  on  the  benefits  of  virtual  mission 
preparation  from  the  fighter  aircrew  roleto  that  of  command  and  control  teams;  specifically, 
Air  Battle  Management  (ABM)  Teams  from  41  Wing,  Surveillance  and  Response  Group.  These 
teams  are responsibleforthetacti cal  command  and  control  of  all  ai  r  assets  in  the  battlesp ace, 
and  are  typically  comprised  of  aTactical  Director,  and  a  number  of  Fighter  Controllers.  The 
T acti  cal  D  i  rector  al  I  ocates  assets  and  manages  operati  ons  w  i  thi  n  the  ai  r  battl  e,  overseei  ng  and 
directing  the  Fighter  Controllers,  and  communicates  with  other  command  elements  and 
external  agencies.  The  Fighter  Controllers  liaise  with  pilots  in  order  to  direct  aircraft  in 
accordance  with  instructions,  procedures,  the  tactical  plan,  and  as  directed  by  theTactical 
Director.  ABM  Teams  deal  with  complex  information  from  a  variety  of  sources,  make 
decisions  in  uncertain  and  dynamic  situations,  and  have  distinct  but  interdependent  roles. 
These  features  of  their  role  make  them  representative  of  command  and  control  teams  more 
generally.  Third,  weemploy  measures  relating  to  the  performance  of  mission  essential  tasks 
aswell  as  measures  relating  to  generic  teamwork  processes  and  motivational  states  critical  to 
effective  team  performance:  communication,  information  exchange,  supporting  behaviour, 
initiative  and  leadership,  collective  efficacy,  and  cohesion  (llgen,  FI ol I enbeck,  Johnson,  & 
Jundt,  2005;  Kozlowski  &  llgen,  2006;  Salas,  Sims,  &  Burke,  2005).  Finally,  by  employing  a 
matched  control  group  that  received  no  formal  training,  this  study  permits  a  more  rigorous 
evaluation  of  the  absolute  effectiveness  of  virtual  mission  preparation  than  permitted  by 
previous  studies. 

1.2.1  Research  Questions 

Asstated  previously,  theprinci  pal  aim  of  this  study  was  toinvesti  gate  the  potential  benefits 
of  using  synthetic  environments  to  provide  mission  preparation  prior  to  a  live  warfighting 
exercise.  Within  this  principal  aim  are  several  distinct  research  questions  that  permit  an 
evaluation  of  the  training  provided  in  Exercise  Black  Skies  according  to  Kirkpatrick's  (1959) 
criteria: 

Reactions  Criteria: 

1.  What  opinions  do  members  of  RAAF  ABM  Teams  hold  about  the  utility  and 
applications  of  virtual  mission  preparation,  and  Exerci se  Bl ack  Skies  in  particular? 

2.  Flow  do  synthetic  training  environments  such  as  the  one  employed  in  Exercise  Black 
Skies  compare  to  live  exercises,  and  regular  training,  in  the  capacity  to  provide 
experiences  critical  to  the  ABM  role? 

Learning  Criteria: 

3.  What  i  mpact  does  parti ci  pati  ng  i  n  team  trai  ni  ng  i  n  a  syntheti c  envi  ronment  have  on 
key  teamwork  processes  and  motivational  states? 

4.  What  i  mpact  does  parti  ci  pati  on  i  n  team  trai  ni  ng  i  n  a  syntheti  c  envi  ronment  have  on 
the  performance  of  mission-essential  tasks? 
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Behavioural  Criteria: 

5.  Do  ABM  Teams  that  take  part  in  virtual  mission  preparation  demonstrate  more 
effecti  veteam  processes  duri  ng  I  i  ve  warfighti  ng  exerci  ses  than  teams  that  prepare  by 
supporting  normal  flying  operations? 

6.  Do  ABM  Teams  that  take  part  in  virtual  mission  preparation  perform  bdter  in  live 
warfighting  exercises  than  teams  that  prepare  by  supporting  normal  flying 
operations? 

Results  Criteria: 

7.  What  broader  value  is  there  for  the  RAAF  in  providing  command  and  control  teams 
with  virtual  mission  preparation  prior  to  live  warfighting  exercises? 

A  subset  of  these  research  questions  can  be  associated  with  research  hypotheses.  Specifically, 
w  e  pred  i  ct  that  the  trai  ni  ng  provi  ded  i  n  Exerci  se  Bl  ack  Ski  es  w  i  1 1  resul  t  i  n  an  i  improvement  i  n 
teamwork  processes,  motivational  states,  and  the  performance  of  tasks  critical  to  the  ABM 
team  role.  Furthermore,  wepredict  that  thelearning  that  occurs  over  thecourse  of  thevirtual 
mi  ssi  on  preparati  on  w  i  1 1  transfer  to  the  I  i  ve  warf  i  ghti  ng  envi  ronment,  w  i  th  the  resul  t  that  the 
ABM  team  that  received  thevirtual  mission  preparation  will  perform  better-on  measures  of 
both  taskwork  and  teamwork  -  than  the  ABM  team  that  prepared  by  supporting  normal 
flying  operations. 


2.  Method 


2.1  Design  and  Participants 

2.1.1  Design 

A  case  study  strategy  was  employed  to  examinethe  benefits  of  using  synthetic  environments 
to  provide  mission  preparation  for  command  and  control  teams  prior  to  a  large-scale,  live 
warfighting  exercise.  The  live  exercise  used  as  a  vehicle  for  this  study  was  Pitch  Black  08,  a 
biennial  combined  air  and  ground  training  exercise  hosted  by  the  RAAF  and  involving 
participants  from  a  number  of  foreign  military  forces  (Department  of  Defence,  2009).  Pitch 
Bl  ack  08  took  pi  ace  arou  nd  D  arw  i  n  and  Ti  nd  al  i  n  J  u  ne,  2008,  and  the  broad  ai  m  of  the  exerci  se 
was  to  test  and  improve  the  command  and  control  in,  and  execution  of,  a  multi-national 
coalition  air  campaign. 

OneABM  team  -  the  control  team-  prepared  for  Pitch  Black  08  by  supporting  normal  flying 
operations,  as  well  as  taking  part  in  additional  training  tailored  specifically  for  the  live 
exercise.  This  tailored  preparation  commenced  two  weeks  prior  to  Pitch  Black  08,  and 
consisted  predominantly  of  practice  mi  ssi  onswhich  involved  control  ofWilliamtown  aircrew 
i  n  smal  I  er  seal  e  mi  ssi  ons  than  those  seen  at  Pitch  Bl  ack  08.  The  mi  ssi  ons  general  I  y  escal  ated  i  n 
difficulty  overthework  up  period,  but  did  not  reach  thecomplexity  or  difficulty  of  the  Pitch 
Black  08  missions.  During  this  work-up  period  the  control  ABM  team  attended  pre-mission 
briefsandpost-missiondebriefswiththeai  rcrew  .Thisteamwerealso  asked  to  read  th  e  P  i  tch 
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Black  08  exercise  documentation,  which  included  the  airspace  control  plan,  the  airspace 
d  ef  ence  p  I  an ,  and  the  ai  r  su  rvei  1 1  ance  p  I  an .  The  members  of  the  team  w  ere  then  requ  i  red  to  si  t 
atheoryexamtoensureall  documentation  was  understood.  A  second  ABM  team  prepared  for 
Pitch  Black  08  by  taking  part  in  Exercise  Black  Skies  08.  The  synthetic  environment  provided 
in  Black  Skies  08  was  designed  to  simulate  Pitch  Black  08  as  closely  as  possible  in  terms  of 
mission  scenarios,  order  of  battle,  airspace,  procedures,  and  tactics.  A  description  of  the 
mission  scenarios  is  presented  in  Section  2.3.1. 

As  stated  in  the  Introduction,  the  evaluation  of  thetraining  provided  in  the  virtual  mission 
preparation  -  Black  Skies  08  -  was  carried  out  using  thefour  levels  of  criteria  suggested  by 
Kirkpatrick  (1959):  Information  was  gathered  about  theABM  team's  reactions  to  the  virtual 
mission  preparation,  the  learning  that  occurred  over  the  course  of  the  virtual  mission 
preparation,  theimpact  of  thevirtual  mission  preparation  on  performance  in  the  subsequent 
live  warfighting  exercise1,  and  the  broader  value  to  the  organisation  of  thevirtual  mission 
preparation. 

2.1.2  Participants 

Two  ABM  Teams  from  41  Wing(WG),  Surveillance  and  ResponseGroup  (SRG),  participated 
i  n  the  study.  Each  team  consisted  of  a  Tactical  Di  rector,  and  three  Fighter  Control  I  ers  i  n  direct 
control  roles,  all  of  whom  possessed  operational  experience. 

The  Fighter  Controllers  were  allocated  to  either  the  virtual  mission  preparation  or  control 
conditions  by  41WG  in  a  quasi-random  manner;  random  allocation  was  constrained  by  the 
desire  to  produce  ABM  Teams  that  were  matched  in  terms  of  the  operational  experience  of 
thei  r  members,  and  by  the  avai  lability  of  individual  Fi  ghter  Control  I  ers  for  the  vi  rtual  mi  ssi  on 
preparation  phase  of  the  study. 

2.1.3  White  Force  and  Assessor 

The  execution  of  the  virtual  mission  preparation  was  managed  by  aWhiteForceconsistingof 
a  White  Force  M  i  ssi  on  Di  rector,  a  Red  si  mulation  operator  (SI  M  OP)  coordinator,  five  SI  M  OPs, 
and  an  Air  Battle  Director.  The  role  of  the  White  Force  Mission  Director  was  contracted  to 
Milskil  Integrated  Solutions  (the  company  responsible  for  developing  and  managing  the 
execution  of  the  Pitch  Black  08  vignettes),  and  was  filled  by  an  ex-RA  A  F  fighter  pilot.  TheAir 
Battle  Director  and  the  five  SIM  OPs  were  members  of  41  WG  SRG,  and  the  Red  SI  MOP 
coordinatorwasaDSTO  staff  member  who  possessed  operational  experience  as  an  ABM  with 
the  RA  A  F.  The  Bl  ue  SI  M  O  Ps  mani  pul  ated  the  Bl  ue  Force  ai  r  assets  under  the  di  recti  on  of  the 
White  Force  Mission  Director.  The  Red  SIM  OPs,  under  the  direction  of  the  Red  SI  MOP 
coordinator,  were  responsible  for  both  manipulating  the  synthetic  fighter  assets  under  the 
control  of  theABM  team  and  simulating  the  communications  of  those  aircraft  pilots.  In  both 
Black  Skies  08  and  Pitch  Black  08,  the  evaluation  of  taskwork  performance  and  teamwork 


1  For  reasons  of  both  safety  (e.g.,  conducting  research  in  a  live-control  environment)  and  resources  (e.g., 
access  to  an  appropriate  assessor)  we  were  only  able  to  assess  the  performance  of  the  ABM  Teams  on  the  first 
two  days  of  Pitch  Black  08. 
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processes  was  carried  out  by  an  assessor  from  Surveillance  and  Control  Training  Unit 
(SACTU),  41WG  SRG. 

2.2  Measures 

2.2.1  Measures  of  Trainee  Reactions  to  the  Virtual  Mission  Preparation 

22.1.1  Fighter  Controller  Reactions  Survey 

General  opi  ni  ons  about  mi  ssi  on  trai  ni  ng  through  d  i  stri  buted  si  mu  I  ati  on,  and  reacti  ons  to  the 
training  provided  in  Black  Skies  08  in  particular,  were  assessed  using  the  Fighter  Controller 
Reactions  Survey.  This  58-item  survey  was  adapted  for  the  RAAF  ABM  context  from  a 
questionnaire  developed  by  AFRL  and  DSTL;  this  questionnaire  was  designed  to  be  used  by 
AWACS  controllers  -  who  perform  a  similar  function  to  RAAF  Fighter  Controllers  -  for  the 
purposes  of  the  evaluation  of  coalition  mission  training  research  exercises  (Smith,  dial.,  2006). 
Participants  used  a  7-point  scale(l  =strongly  disagree,  7  =strongly  agree)  to  rate  the  extent  to 
which  they  agreed  with  statements  such  as  'EBS  08  has  helped  me  improve  my  team 
coordination  skills'  and  'mission  training  through  distributed  simulation  provides  realistic 
training  and  experience.' 

2.2.12  Training  Environments  Survey 

The  rel  ati  ve  uti  I  i  ty  of  vi  rtual  exerci  ses  as  an  envi  ronment  i  n  w  hi  ch  to  trai  n  experi  ences  cri  ti  cal 
to  the  ABM  role  was  assessed  using  the  Training  Environments  Survey.  The  survey,  also 
adapted  from  a  questionnaire  developed  by  AFRL  and  DSTL  for  the  evaluation  of  coalition 
mission  training  research  exercises  (Smith,  et  al„  2006),  contains  66  items  representing 
experi  ences  commonly  encountered  in  RAAF  ABM  team  mi  ssi  ons.  Parti  ci  pants  used  a  5-point 
seal  e(l=notatall,5=toa  great  ex  ten  t)  to  i  nd  i  cate  the  extent  to  w  hi  ch  i  t  i  s  reason  abl  y  possi  bl  e 
to  provide  each  experience  in  each  of  three  different  training  environments:  (1)  virtual 
exercises  such  as  Black  Skies  08,  (2)  live  exercises  such  as  Pitch  Black  08,  and  (3)  training  at 
thei  r  home  unit.  Thefi  rst  two  envi  ronments  have  been  descri  bed  previ  ously.  Thetrai  ni  ng  that 
ABM  teams  receive  at  their  home  unit  consists  largely  of  the  individual  training  of  Fighter 
Controllers  -  both  live  and  simulated  -  in  procedures  and  tactics  critical  to  the  Air  Battle 
M  anagement  rol  e.  A  BM  T earns  actual  I  y  rarel  y  trai  n  as  a  team  pri  or  to  depl  oy  ment,  asi  defrom 
that  training  offered  in  live  exerci  ses  such  as  Pitch  Black. 

2.2.2  Motivational  State  Measures 

2.22.1  Collective  Efficacy 

The  col  I  ecti  ve  effi  cacy  of  theteam  that  recei  ved  the  vi  rtual  mi  ssi  on  preparati  on  was  assessed 
using  an  8-item  scale  that  focused  on  the  individual's  perception  of  the  team's  ability  to 
effecti  vel  y  cope  w  i  th  the  chal  I  enges  of  thei  r  task.  Parti  ci  pants  were  asked  to  rate,  on  a  7-poi  nt 
scale(l  =strongly  disagree,  7  =strongly  agree),  theextentto  which  they  agreed  with  statements 
such  as  'this  team  can  meet  the  challenges  of  the  task'  and  'this  team  will  cope  if  the  task 
becomes  morecomplex.'  The  measurewas  adapted  from  a  scale  developed  by  Kozlowski  et 
al.  (2001)  and  described  more  comprehensively  by  DeShon  etal.  (2004),  who  reported  a  one- 
factor  solution  with  coefficient  alphas  ranging  from  .95  to  .96  over  three  successive 
administrations. 
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2.2.22  Cohesion 

The  cohesion  of  theteam  that  received  the  virtual  mission  preparation  was  assessed  using  a  6- 
item  scalefocusing  on  the  extent  to  which  theteam  members  work  together  to  accomplish 
their  task.  Participants  rated,  on  a  7-point  scale  (l=strongly  disagree,  7  =strongly  agree),  the 
extent  to  which  they  agreed  with  statements  such  as  'themembers  of  thisteam  work  hard  to 
get  things  done'  and  'themembers  of  thisteam  make  each  other  feel  like  doing  a  good  job.' 
The  measu re  w as  ad apted  from  Si  ebol  d  and  K el  I y 's  ( 1988)  i  nstru mental  cohesi  on  seal  e,  w i th  a 
reported  coefficient  alpha  of  .83. 

2.2.3  Taskwork  Performance  Measure 

Taskwork  performance  was  assessed  against  a  hierarchical  set  of  measures  designed  to 
capture  all  the  mission-essential  tasks  required  of  an  ABM  team  in  conducting  a  Defensive 
Counter  Air  operation  (Hasenbosch  &  Best,  2007),  focusing  only  on  those  aspects  of  the  task 
that  rely  on  teamwork.  The  hierarchy  contains  six  highest-level  categories  (develop  and  brief 
mission  plan,  establish  military  liaison,  manage  information  systems,  control  air  space, 
conduct  defensive  counter  air,  and  protect  key  points  and  vital  assets)  and  47  items  at  the 
lowestlevel  of  measurement.  Thesix  highest-level  categories  are  I  inked  tothetactical  level  of 
the  Australian  Joint  Essential  Tasks  List  (McCarthy  et  al„  2003);  this  list  presents  a 
comprehensivedescription  of  all  tasks  essential  to  the  preparation,  planning,  and  conduct  of 
Australian  Defence  Force  (ADF)  operations.  The  system  of  performance  measurement 
employed  in  this  study  enables  commanders  to  identify  the  strengths  and  weaknesses  of 
teams  in  the  broader  context  of  required  operational  capability.  The  assessor  used  appoint 
anchored  scale  (1  =terrible,  2  =inferior,  3  =passable,  4  =good,  5  =excellent;  taken  from  Spector, 
1992)  to  rate  the  performance  of  the  ABM  team  against  the  behavioural  criterion  attached  to 
each  lowestlevel  item. 
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T  ablel:  Behavioural  anchors  for  the  scale  used  to  assess  the  four  dimensions  of  teamwork  processes 


Poor  Behaviours 

Excellent  Behaviours 

Communication 

Team  members  consistently  demonstrated 
improper  use  of  phraseology,  excess  chatter, 
i  ncomp  1  ete  reports,  i  nau  d  i  bl  e  commu  n  i  cati  on 

Team  members  displayed  no  instances  of 
improper  phraseology,  excess  chatter,  incomplete 
reports,  inaudiblecommunication 

Information  Exchange 

Team  members  failed  to  use  information  from  all 
available  sources,  did  not  pass  on  information 
before  having  to  be  asked,  did  not  provide 
situation  updates 

Team  members  madeuseof  information  from  all 
avail  able  sources,  passed  on  information  without 
having  to  be  asked,  provided  good  situation 
updates 

Supporting  Behaviour 

Team  members  did  not  monitor  one-another  for 
errors,  or  take  action  to  correct  errors  when  they 
occurred.  They  did  not  request  or  offer  assistance 
when  it  was  required  to  adjust  workload  within 
the  team 

Team  members  consistently  monitored  one- 
another  for  errors  and  took  action  to  correct  errors 
when  they  occurred.  They  requested  and  offered 
assistance  when  required  to  adjust  workload 
within  the  team 

Initiative  and  Leadership 

Team  members  did  not  offer  guidance  or 
suggestions  to  others  when  required.  They  were 
unclear  about  their  own,  and  the  team's 
priorities,  or  failed  to  state  these  cl  early 

Team  members  offered  guidance  and  suggestions 
to  others  when  required.  They  understood  their 
own,  and  the  team's  priorities  and  stated  these 
clearly 

2.2.4  Teamwork  Processes  Measure 

Teamwork  processes  were  assessed  on  a  scale  comprised  of  the  four  principal  dimensions 
identified  by  Smith-Jentsch,  Zeisig,  Acton,  and  McPherson  (1998):  communication, 
i  nformati on  exchange,  su pporti  ng  behavi our,  and  i  ni ti ati  ve and  I eadershi  p.  Thesed i  mensi ons 
were  identified  by  Smith-Jentsch  etal.,  (1998)  from  an  analysis  of  performance  data  from  US 
Navy  combat  information  centreteams.  Smith-Jentsch  etal.  included  only  thefactors  which 
weredi  redly  observableand  also  involved  team  performance (i.e.,  they  excluded  individual- 
I  evel  behavi  ours).  These  d  i  mensi  ons  and  si  i  ght  vari  ati  ons  thereof  have  been  w  i  del  y  uti  I  i  sed  i  n 
the  team  training  literature(e.g.,  McIntyre  and  Salas,  1992;  Serfaty,  Entin  &  Johnston,  1998; 
Smith-Jentsch, Johnston  &  Payne,  1998;  Rousseau,  Aube&  Savoie,  2006).  Participants  (and  the 
assessor)  wereasked  to  ratethequal  ity  of  teamwork  processes  usi  ng  a  7-poi  nt  scale2  anchored 
with  descri  pti  ons  of  poor  and  excel  I  ent  behavi  ours  rel  ated  to  each  di  mensi  on;  the  behavi  oural 
anchors  are  I  i  sted  i  n  Tabl  e  1. 


2  In  Pitch  Black  08,  the  assessor  employed  a  5-point  scale  -  incorporating  the  same  behavioural  anchors  as  the 
7-point  scale  -  to  rate  teamwork  processes.  For  ease  of  comparison  with  Black  Skies  08  data,  these  ratings 
were  transformed  to  a  7-point  scale. 
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2.3  The  Virtual  Mission  Preparation  Environment 

The  virtual  mission  preparation  provided  for  the  ABM  team  in  Black  Skies  08  was  designed  to 
match,  as  cl  osely  as  possi  ble,  theenvi  ronment  i  n  which  they  would  work  and  thetasks  which 
they  would  berequired  to  perform  during  Pitch  Black  08.  A  comprehensivedescription  ofthe 
physical  environment  and  simulation  infrastructure  employed  in  the  virtual  mission 
preparation  phase  of  the  study  is  provided  in  the  companion  report  (Tracey  etal.,  2009);  here, 
we  provide  a  brief  overview  of  the  mission  scenarios,  management,  physical  environment, 
information  systems,  and  scenario  generation  in  Black  Skies  08. 

2.3.1  M  ission  Scenarios  in  Exercise  Black  Skies  08  and  Pitch  Black  08 

Pitch  Black  08  was  comprised  of  a  series  of  mission  scenarios  (or  vignettes )  which,  over  the 
course  of  theexercise,  portrayed  an  escalation  intension  between  the  opposing  Blueand  Red 
forces  and  a  progression  of  military  action.  The  vignettes  varied  in  terms  of  training 
objectives,  order  of  battle,  rules  of  engagement,  airspace,  threats,  and  targets;  they  did  not 
differ  in  terms  of  difficulty  or  complexity.  Broadly  speaking,  the  Blue  Force  participants  were 
required  to  conduct  a  variety  of  missions  such  as  offensive  counter  air  (OCA),  offensive  air 
support  (OAS),  deep  stri  ke,  destruction  of  enemy  ai r  defences,  and  joi  nt  personnel  recovery; 
the  Red  Force  parti  ci  pants  responded  by  empl  oyi  ng  defensive  counter  ai  r  (DCA )  tacti  cs.  The 
41WG  ABM  Teams  worked  in  shifts  to  provide  tacti  cal  command  and  control  support  to  the 
Red  Force. 

Of  the  seven  vignettes  developed  for  Pitch  Black  08,  four  were  used  in  the  virtual  mission 
preparation  provided  in  Black  Skies  08:  Vignette  A,  involving  a  Blue  Force  OCA  mission 
(Day  lof  EBS  08);  VignetteB,  involving  BlueForceOCA  and  strike  (Days  2  and  5);  VignetteD, 
involving  a  Blue  Force  joint  personnel  recovery  mission  (Day  3);  and  Vignette  E,  which 
involved  a  Blue  Force  deep  strike  mission  (Day  4).  In  each  Black  Skies  08  vignette,  between  40 
and  60  Blueand  Red  air  entities  were  generated. 

2.3.2  Physical  Environment  in  Exercise  Black  Skies  08 

In  Pitch  Black  08,  the  ABM  teamoperated  outoftheTactical  Control  Centre,  Eastern  Regional 
Operations  Centre,  at  RAAF  Williamtown.  For  Black  Skies  08,  a  scaled-down  version  of 
Tactical  Control  Centre  was  constructed  in  the  Air  Operations  Experimentation  Centre 
(AOEC)  at  DSTO  Melbourne.  The  layout  of  theAOEC  during  Black  Skies  08  is  depicted  in 
Figures  1  and  2:  Figure  1  showsthe  positions  of  members  of  theABM  team  and  the  White 
Force,  and  Fi  gure  2  show s  the  vi  ew  from  the  stati  on  of  theT acti  cal  D  i  rector,  i  n  front  of  w  hi  ch 
are  the  stations  of  the  three  Fighter  Controllers  under  his  command. 
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Observer  Space 


Tactical  Control  Centre 


Fighter  Controller 
3 


Fighter  Controller 
2 


Fighter  Controller 
1 


White  Force  Control  Centre 


Figure  1:  A  top-down  representation  of  the  simulation  facility,  showing  the  control  cells  and 
positioning  of  personnel 


:ical  Director  Workstation 


Projection  1 


Projection  2 


FC2  Workstation 


FC3  Workstation 


Figure2:  Thetactical  control  centre,  showing  theposition  of  members  of  theTacti  cal  Director  and  the 
three  Fighter  Controllers  (FC),  the  layout  of  operator  workstations,  and  the  visual  tools 
used  during  after  action  review  (the  SM  ART  Board  and  projection  surfaces) 
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2.3.3  Information  Systems  in  Exercise  Black  Skies  08 

The  ABM  team  was  provided  with  information  systems  which  faithfully  replicated  the 
operational  command  and  control  system  used  by  the  RAAF:  the  Soli  psys  tactical  situation 
display  software,  a  simulation  of  the  radio  and  intercom  equipment,  and  theToteboard 
application  (used  to  collect  and  disseminate  information  aboutthefuel  and  weapon  status  of 
ai  rcraft). 

2.3.3.1  Scenario  Generation  in  Exercise  Black  Skies  08 

TheWhite  Force  used  Pi  lot  Simulation  Interface  software  to  create  and  control  entities.  Each 
instance  of  PSI  was  connected  to  a  server  operating  theVR-Forces  software,  which  modelled 
al  I  the  ai  rcraft  i  n  the  si  mu  I  ati  on  envi  ronment  and  broadcast  the  i  nformati  on  vi  a  a  D  i  stri  buted 
Interactive  Simulation  (DIS)  protocol. 

An  Air  Defence  Ground  Environment  Simulator  (ADGESIM)  server  application  called 
SensorLink  was  used  to  model  the  Pitch  Black  08  radar  types  and  sites,  and  to  translate  the 
detection  of  entities  received  via  the  DIS  protocol  to  a  plot  protocol  compatible  with  the 
multiple  source  correlator/  tracker. 

The  multi  pie  source  correlator/  tracker  was  connected  to  each  instance  of  the  Soli  psys  tactical 
situation  display  software;  this  software  was  configured  with  airspace  and  waypoint  data, 
and  a  RAAF-developed  user  profile  which  defines  aspects  such  as  data  block  fields,  identify 
friend  or  foe  (IFF)  decoding,  and  a  multitude  of  appearance  settings. 

2.4  Virtual  Mission  Preparation  Procedure 

The  virtual  mission  preparation  took  pi  ace  two  weeks  prior  to  the  commencement  of  Pitch 
Black  08.  On  thefirst  day  of  the  virtual  mission  preparation  the  ABM  team  was  briefed  on  the 
purpose  and  goal  s  of  the  exerci  se,  the  exerci  se  scenari  o,  the  manner  i  n  w  hi  ch  thei  r  taskwork 
performance  and  teamwork  processes  would  be  evaluated,  and  thescheduleof  events.  They 
completed  theteam  cohesion,  collective  efficacy,  and  teamwork  processes  measures,  and  were 
then  allocated  time  to  familiarise  themselves  with  the  setup  of  the  command  and  control 
systems. 

One  mission  was  run  on  each  day  of  Black  Skies  08,  beginning  on  the  afternoon  of  thefirst 
day.  Theproceduresurrounding  each  mission  consisted  ofthefollowing  key  events:  scenario 
update,  mission  preparation,  mission  execution,  measurement  session,  after  action  review 
(AAR),  and  exercise  feed  back3. 

In  the  scenario  update,  the  ABM  team  were  briefed  on  expected  threats,  order  of  battle,  rules 
of  engagement  (ROE),  ai  rspace,  and  enemy  i  ntent.  The  preparati  on  ti  me  was  used  to  pi  an  for 
the  impending  mission.  In  each  mission,  the  ABM  team  was  required  to  command  the  air 
assets  defending  Red  airspace  and  key  points  against  the  larger  and  technologically  superior 


3  The  instability  of  the  simulation  system  necessitated  minor  modifications  to  the  schedule;  see  Tracey  et  al. 
(2009)  for  greater  detail. 
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Blue  Force.  Their  taskwork  performance  and  teamwork  processes  were  assessed  by  the 
SACTU  assessor  and  the  Air  Battle  Director  respectively;  the  taskwork  assessment  was 
completed  using  a  semi -automated  performance  assessment  and  feedback  tool  developed  by 
DSTO  in  partnership  with  theRAAF  (see  Tracey  etal.,  2009,  for  a  description  of  this  tool).  In 
the  measurement  session  following  each  mission,  the  ABM  team  completed  the  team 
cohesi  on,  col  I  ecti  ve  effi  cacy,  and  teamwork  processes  measures.  I  n  the  A  A  R  sessi  on,  the  A  BM 
team  received  structured  feedback  about  the  effectiveness  of  thei  r  teamwork  processes,  and 
thei  r  performance  was  eval  uated  agai  nst  thespecifi  c  trai  ni  ng  objectives  contai  ned  i  n  theA  BM 
DCA  hierarchy.  This  performance  feed  back  was  supported  by  replays  of  the  ground  truth 
synchronised  with  replays  of  the  tactical  displays  and  communications.  In  the  exercise 
feedback  sessi  ons  theA  BM  team  compl  eted  the  Fi  ghter  Control  I  er  Reacti  ons  Survey  (Day  4), 
theTraining  Environments  Survey  (Day  5),  took  part  in  structured  interviews  relating  to  the 
uti  I  i  ty  of  the  si  mu  I  ati  on  tool  sand  trai  n  i  ng  method  s  emp  I  oyed  i  n  Bl  ack  Ski  es  08,  and  provi  d  ed 
general  feedback  about  the  exercise. 

2.5  Exercise  Pitch  Black  Procedure 

Duri  ng  thefi  rst  week  of  Pitch  Black  the  A  BM  teams  conducted  work  up  mi  ssi  ons  as  trai  ni  ng 
for  the  exerci  se.  Pri  or  to  the  fi  rst  day  of  fl  y  i  ng  there  was  a  M  ass  A  i  r  Bri  ef  i  n  w  hi  ch  al  I  exerci  se 
participants  were  familiarised  with  the  specifics  of  the  exercise  including  the  scenarios, 
airspace,  procedures,  and  manning  requirements.  During  the  exercise  the  ABM  teams 
controlled  onedefensivecounter  air  mission  aday.  These  mi  ssi  ons  were  preceded  with  abrief 
by  theTactical  Director.  The  Tactical  Director  briefed  the  aircrew  using  video  conferencing 
and  then  briefed  the  ABM  team.  The  ABM  team  brief  consisted  of  issues  such  as 
communications  requirements,  airspace  layout  and  restrictions  and  any  pointswhich  came 
out  of  the  bri  ef  w  i  th  the  ai  rcrew .  Fol  I  ow  i  ng  mi  ssi  on  executi  on  theT acti  cal  D  i  rector  debri  efed 
the  A  BM  team  and  then  the  ai  rcrew . 


3.  Results 

The  results  are  reported  within  the  framework  of  Kirkpatrick's  (1959)  four  criteria  for  the 
evaluation  of  training  programs:  Reaction,  Learning,  Behavioural,  and  Results.  In  the  first 
secti  on,  we  report  theA  BM  team's  general  i  mpressi  on  of  Bl  ack  Ski  es  08  as  a  trai  ni  ng  acti  vi  ty, 
as  well  as  their  bel i efs  about  the  broad  and  specific  utility  of  virtual  mission  preparation.  In 
the  second  section,  we  report  the  extent  to  which  Black  Skies  08  resulted  in  improvements  in 
theABM  team's  taskwork  performance,  teamwork  processes,  and  key  motivational  states.ln 
the  third  section,  the  Pitch  Black  08  performance  of  theABM  team  that  received  the  virtual 
mission  preparation  is  compared  to  that  of  a  team  that  prepared  by  supporting  normal  flying 
operations.  In  thefi  nal  section,  we  highlight  someof  the  ways  in  which  Black  Skies  08  was  of 
considerable  organisational  value  to  the  RAAF. 


15 


DSTO-RR-0344 


3.1  Reaction  totheTraining  Provided  in  theVirtual  Mission  Preparation 

3.1.1  Fighter  Controller  Reactions  Survey 

The  mean  ratings  of  ABM  team  member  agreement  with  all  items  in  the  Fighter  Controller 
Reactions  Survey  are  presented  in  TableAl,  Appendix  A.  Only  the  more  salient  results,  by 
category,  will  be  elaborated  in  this  section. 

3. 1.1.1  Overall  U  tility  and  Potential  A  pplications 

Atabroad  level,  members  of  the  ABM  team  viewed  Black  Skies  08  as  a  good  use  of  their  time, 
and  would  recommend  the  experience  to  other  controllers.  They  agreed  virtual  mission 
preparati  on  of  thi  s  ki  nd  had  consi  derabl  e combat  mi  ssi  on  trai  ni  ng  val  ue,  and  that  i t  shou I  d  be 
employed  as  both  spin-up  for  live  exercises  and  as  part  of  their  readiness  training  schedule. 

3. 1.1.2  Impact  on  M  ission  Skills  and  Readiness 

Si  gnif  i  cantl  y ,  the  A  BM  team  reported  that  Bl  ack  Ski  es  08  had  i  mproved  thei  r  combat  mi  ssi  on 
readiness,  and  that  they  would  perform  better  as  a  controller  as  a  result  of  Black  Skies  08. 
More  specifically,  the  ABM  team  felt  that  as  a  result  of  the  virtual  mission  preparation  they 
had  improved  their  team  coordination  skills,  combat  situational  awareness,  tactical  skills,  and 
were  better  prepared  to  brief  and  lead  a  mission.  However,  they  were  ambivalent  as  to 
whether  they  had  improved  their  mission  planning  skills,  understanding  of  critical  combat 
skills,  or  debriefing  skills. 

3. 1.1.3  Specific  Training  Opportunities 

The  ABM  team  agreed  that  Black  Skies  08  provided  valuable  experience  across  a  number  of 
areas  that  are  critical  to  conducti  ng  the  DC  A  mission,  i  nd  udi  ng:  i  nteracti  ng  with  a  variety  of 
people  and  systems,  team  coordination,  multiple  complex  presentations,  and  large  force 
control.  In  fact,  the  ABM  team  felt  that  Black  Skies  08  provided  a  training  experience  that  is 
not  avail  able  at  their  home  unit;  as  a  wholetheteam  was  equivocal  as  to  whether  their  home 
unitroutinely  provided  theopportunity  to  practice  tacticsin  multi  pie  aircraft  environments, 
or  to  control  aircraft  against  realistic  threats.  Although  the  ABM  team  agreed  strongly  that 
when  they  returned  to  their  operational  unitthey  would  havetheopportunitytousetheskills 
learned  during  Black  Skies  08,  they  were  unsure  whether  theoperations  tempo  at  their  home 
unit  would  help  to  maintain  the  skills  learned  during  Black  Skies  08. 

3. 1.1.4  Relation  between  Virtual  M  ission  Preparation  and RegularTraining 

O  veral  I ,  the  A  BM  team  agreed  that  Bl  ack  Ski  es  08  provi  ded  real  i  sti  c  trai  ni  ng  and  experi  ence. 
Importantly,  the  ski  I  Is  trained  in  Black  Skies  08  were  considered  to  be  the  same  as  those  the 
ABM  team  required  in  the  field.  However,  the  ABM  team  were  equivocal  about  both  the 
extent  to  which  the  exercise  envi  ronment  itself  was  a  real  i sti c  representation  of  combat,  and 
whether  the  missions  and  engagements  accurately  represented  how  things  happen  in  the 
operational  world. 
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A.  Virtual  Exercises  versus  Live  Exercises 
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Figure  3:  Frequency  histograms  showing  the  distribution  of  differences  between  suitability  ratings  for 
virtual  exercises  versus  live  exercises  (Panel  A)  and  home-unittraining  (Panel  B)  on  a  set 
of  ABM  training  tasks.  Difference  scores  are  grouped  into  half-scale-point  categories. 
Positive  scores  indicatethat  virtual  exercises  were  rated  as  more  suitable  environments  for 
given  tasks,  negative  scores  indicatethat  that  virtual  exercises  were  rated  as  I  ess  suitable. 

31.1.5  Training  Design  and  Procedure 

From  a  system  perspective,  the  scenarios  were  viewed  as  well  designed  with  clear  learning 
objectives.  Although  the  missions  and  engagements  were  considered  to  be  challenging,  and 
helped  the  ABM  team  meet  these  learning  objectives,  they  had  no  difficulty  in  keeping  up 
with  the  pace  of  the  missions. 

3.1.2  Training  Environments  Survey 

Overall,  virtual  exercises  such  as  Black  Skies  08  compared  favourably  to  more  established 
trai  ni  ng  envi  ronments  i  n  terms  of  the  abi  I  ity  to  provi  de  experi  ences  criti  cal  to  the  A  BM  role. 
The  frequency  histograms  in  Figure  3  show  how  the  ABM  team  that  participated  in  Black 
Skies  08  rated  the  relative  suitability  of  virtual  exercises,  live  exercises,  and  home-unittraining 
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as  environments  for  training  a  given  set  of  ABM  tasks4.  Panel  A  shows  the  differences 
between  suitability  ratings  given  to  virtual  exercises  and  live  exercises.  Panel  B  shows  the 
differences  between  suitability  ratings  given  to  virtual  exercises  and  home  unit  training. 
Difference  scores  aregrouped  into  half-scale-point  categories  in  order  to  form  thefrequency 
histogram.  Positive  scores  indicate  that  virtual  exercises  were  rated  as  more  suitable 
environments  for  given  training  tasks,  while  negative  scores  indicate  that  virtual  exercises 
were  rated  as  less  suitable.  Panel  A  shows  that  virtual  exercises  were  rated  as  equal  or  more 
suitable  (i.e.,  difference  scores  of  zero  or  greater)  than  live  exercises  for  training  almost  two- 
thirds  of  thetraining  tasks  (62%).  Panel  B  shows  that  virtual  exercises  were  rated  as  equal  or 
moresuitablethan  home-unit  training  for  all  butafew  of  thetraining  tasks  (94%). 

3. 1.2.1  Virtual  Exercises  compared  to  Live  Exercises  asTraining  Environments 
Significantly,  virtual  exercises  were  regarded  as  just  as  suitable  as  live  exercises  for  the 
training  of  core  ABM  team  experiences  such  as  tanker  management  and  air  refuelling, 
protection  of  high  value  airborne  assets,  prosecuting  high-fast-flyers,  and  dynamic  air  combat 
manoeuvring. 

V  i  rtu  al  exerci  ses  w  ere  al  so  vi  ew  ed  as  comparabl  e  to  I  i  ve  exerci  ses  i  n  thei  r  abi  I  i  ty  to  provi  d  e 
experiences  relevant  to  working  with  large  groups  (e.g„  tracking  of  large  Blueand  Red  Air 
groups,  the  safe  control  of  I  arge  fighter  groups  and  strike  packages)  and  working  in  a  difficult 
sensor  and  communications  environment  (e.g. ,  denseinternal  and  external  communications, 
worki  ng  with  a  degraded  pi cture,  thetroubl eshooti  ng  of  pri  mary  and  secondary  sensors,  and 
working  in  a  radar/  IFF/  communications  jamming  environment).  For  the  most  part,  virtual 
exercises  were  seen  as  equivalent  to  live  exercises  in  their  ability  to  provide  experiences 
commonly  part  of  complex  mission  scenarios,  such  as:  various  Rules  of  Engagement,  a  variety 
of  enemy  threats  and  tacti  cs  as  wel  I  as  a  vari  ety  of  fri  end  I  y  capabi  I  i  ti  es  and  tacti  cs,  a  dynami  c 
ground  and  air  enemy  order  of  battle,  rapidly  changing  targets  and  priorities,  and  joint  fires. 

Some  experi  ences  common  to  compl  ex  mi  ssi  on  scenari  os  w  ere  seen  as  more  su  i  ted  to  the  I  i  ve 
exercise  environment,  including:  operating  in  varied  atmospheric  conditions,  force 
marshalling,  and  keeping  track  of  flight  and  time-on-target  deconfliction.  Flowever,  aside 
from  a  small  number  of  assets  and  roles  not  simulated  in  Black  Skies  08  (e.g.,  working  with 
el  ectroni  c  attack  assets,  worki  ng  as  check-i  n  control  I  er),  the  I  argest  category  of  experi  ences  for 
which  virtual  training  environments  were  seen  as  less  suitablethan  live  training  exercises 
related  to  interaction  with  external  agencies:  joint  and  combined  operations  planning,  face  to 
face  mi  ssi  on  planning  (e.g„  with  fighter  aircrew),  Air  Operation  Centrecoordi  nation,  external 
coordination  with  Air  Traffic  Control,  coordination  with  other  C2I SR  players,  and  working 
with  various  ground  parties  (e.g.,  special  operati ons forces,  forward  air  control). 

3. 1.2.2  Virtual  Exercises  compared  to  H  omeUnit  asTraining  Environments 

Only  a  very  small  number  of  experiences  werejudged  bytheABM  team  to  be  more  suited  to 
training  within  their  home  unit  than  within  virtual  exercises  such  as  Black  Skies  08.  These 
included:  acting  as  a  liaison  in  a  contingency  operation  or  exercise,  face  to  face  mission 


4  Table  A2,  Appendix  A,  displays  the  mean  ratings  of  the  extent  to  which  it  was  judged  possible  by  the  ABM 
team  to  provide  critical  experiences  in  virtual  exercises  such  as  Black  Skies  08,  live  exercises  such  as  Pitch 
Black  08,  and  home  unit  training  environments. 
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planning  with  other  elements  (e.g„  fighter  aircrew),  external  coordination  with  Air  Traffic 
Control,  operating  in  varied  atmosphericconditions,  and  hands-on  platform  familiarisation. 

3.2  Learning  over  the  course  of  the  Virtual  M  ission  Preparation 

Over  the  course  of  Exercise  Black  Skies  08,  the  ABM  teamdisplayed  amarked  improvement 
in  taskwork  performance,  teamwork  processes,  and  on  key  motivational  variables.  Outcomes 
pertaining  to  these  aspects  of  performance  are  described  in  detail  below. 

3.2.1  Evaluation  of  Taskwork  Performance  i  n  Exercise  Black  Skies  08 

Summary  ratings  on  the  principal  dimensions  of  taskwork  performance  on  thefirst  and  last 
daysofthevirtual  mission  preparation  aredisplayed  in  Figure4.  Overall,  the  performance  of 
the  ABM  team  ranged  from  passable  to  good.  The  training  appeared  to  have  the  greatest 
impact  on  tasks  associated  with  the  control  of  airspace,  the  management  of  information 
systems,  and  establ  i  shi  ng  mi  I  itary  I  i  ai  son;  thetrai  ni  ng  had  the  I  east  i  impact  on  core  DC  A  tasks, 
whi  ch  i  ncl  udes  factors  such  as  the  management  of  rul  es  of  engagement  and  the  pri  oriti  sati  on 
of  threats. 
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□  Day  1  s  Day  5 


Mission  Establish  Manage  Control  Conduct  Protect  Key 

Planning  and  Military  Information  Airspace  Defensive  Points  and 

Review  Liaison  Systems  Counter  Air  Vital  Assets 


Figure  4:  Summary  ratings  on  the  principal  dimensions  of  taskwork  performance  of  the  team  that 
received  thevirtual  mission  preparation  for  Day  1  and  Day  5  of  Exercise  Black  Skies  08. 
E rror  bars  show  the  range  of  componen  t  performance  ratings.  For  dimensions  where  no 
error  bars  are  displayed  all  component  ratings  were  the  same. 
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A.  Assessor  Rating 
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Communication  Information  Exchange  Supporting  behaviour  Leadership 


Figure 5:  A  ssessor  (Panel  A )  and  mean  self-report  (Panel  B)  ratings  of  team  processes  for  the  team 
that  received  the  virtual  mission  preparation  on  Day  1  and  Day  5  of  Exercise  Black  Skies 
08.  Error  barsin  Panel  B  show  therangeof  self-report  ratings  provided  by  team  members. 
Where  no  error  bars  are  displayed  all  ratings  were  the  same. 

3.2.2  Evaluation  of  Teamwork  Processes  and  Motivational  States  in  Exercise  Black 
Skies  Black  08 

The  ABM  team  demonstrated  a  more  uniform  improvement  in  teamwork  processes  -  as 
opposed  to  taskwork  performance-  over  the  course  of  the  virtual  mission  preparation.  As 
illustrated  in  Panel  A  of  Figure  5,  the  team's  communication,  information  exchange, 
supporting  behaviour,  and  initiativeand  leadership  behaviourswererated  by  the  assessor  as 
more  effective  on  the  last  day  of  Black  Skies  08  than  they  were  on  the  first  day.  For  the  most 
part,  the  ABM  Teams'  ratings  of  their  own  teamwork  processes  indicated  a  similar 
improvement  (Panel  B  of  Figure  5).  As  shown  in  Figure  6,  the  ABM  team  reported  a 
concomitant  increase  in  attitudes  thatfacilitateeffectiveteam  performance-  collective  efficacy 
and  instrumental  cohesion  -  over  the  course  of  Black  Skies  08. 
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Collective  Efficacy  Instrumental  Cohesion 


Figure6:  M  ean  collective  efficacy  and  instrumental  cohesion  scalescores,  for  theteam  that  received 
the  virtual  mission  preparation,  on  Day  1  and  D  ay  5  of  Exercise  Black  Skies  08.  Error  bars 
show  the  range  of  ratings  provided  by  team  members. 

3.3  The  impact  of  Virtual  M  ission  Preparation  on  Performance  in  a  Live 
Warfighting  Exercise 

3.3.1  Evaluation  of  Taskwork  Performance  during  Pitch  Black  08 

Summary  ratings  on  the  principal  dimensions  of  taskwork  performance  for  the  team  that 
received  the  virtual  mission  preparation  and  the  control  team  on  DaysOne(Panel  A)andTwo 
(Panel  B)  of  Pitch  Black  08  are  displayed  in  Figure  7.  Both  teams  performed  at  an  acceptable 
level.  Overall,  how  ever,  the  team  that  received  thevirtual  mission  preparation  outperformed 
theteam  that  prepared  by  supporting  normal  flying  operations.  On  Day  One,  the  greatest 
differences  between  the  teams  were  on  tasks  related  to  establishing  military  liaison, 
management  of  information  systems,  and  protection  of  key  points  and  vital  assets.  On  Day 
Twothegap  in  performance  between  theteams  was  quantitatively  and  qualitatively  different: 
the  gap  was  less  apparent  overall,  and  most  noticeable  on  tasks  related  to  the  control  of 
airspace- these  particular  tasks  were  performed  by  theteams  to  a  roughly  equivalent  level  on 
the  preceding  day. 
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Figure  7:  Summary  ratings  on  the  principal  dimensions  of  taskwork  performance  of  the  team  that 
received  the  virtual  preparation  (T  earn  A )  and  thecontrol  team  (T  earn  B)  for  Day  1  (Panel 
A)  and  Day  2  (Panel  B)  ofPB08.  Error  bars  show  the  range  of  component  performance 
ratings.  For  dimensions  where  no  error  bars  are  displayed  all  component  ratings  were  the 
same 

3.3.2  Quantitative  and  Qualitative  Evaluation  of  Teamwork  Processes  during  Pitch 
Black  08 

As  illustrated  in  Figure  8,  the  team  that  received  the  virtual  preparation  was  rated  by  the 
SACTU  assessor  asdearly  superior  to  thecontrol  team  across  al  I  the  dimensions  of  teamwork: 
information  management,  communication,  supporting  behaviour,  and  initiative  and 
leadership. 
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Figure  8:  A  ssessor  ratings  of  team  processes  for  the  team  that  received  thevirtual  mission  preparation 
and  thecontrol  team  on  Day  1  of  Exercise  Pitch  Black  08 

The  assessor  noted  that  although  there  was  actually  a  higher  frequency  of  communication  in 
the  control  team,  this  communication  was  less  efficient.  For  example,  in  the  control  team, 
communication  between  thefighter  controllers  usually  passed  through  theTacti cal  Director; 
in  the  Black  Skies  team,  thefighter  controllers  communicated  directly,  resulting  in  both  a 
faster  transmi  ssi  on  of  i  nformati  on  and  a  si  gn i f i  cantl y  I  ow er  w orkl  oad  for  theT acti  cal  D i  rector. 
This  pattern  of  communication  is  indicative  of  a  greater  level  of  trust  within  the  ABM  team 
that  received  thevirtual  mission  preparation,  and  greater  ini  tiative  on  the  part  of  the  Fighter 
Controllers. 

Overal  I ,  the  assessor  reported  that  there  was  a  greater  degree  of  i  mpl  i  ci  t  coordi  nati  on  withi  n 
the  Black  Skies  team:  they  were  more  likely  to  provide  information  to  team  members  when 
they  needed  it  and  before  they  had  to  ask  for  it.  They  also  spent  less  time  discussing  the 
team's  tactics  during  mission  execution  and  instead  simply  implemented  previously- 
discussed  plans.  This  suggests  a  greater  familiarity  with  each  other's  roles,  responsibilities, 
and  abilities. 

3.4  The  Benefitto  the  RA  A  F  from  the  Virtual  Mission  Preparation 

Kirkpatrick's  final  training  evaluation  criteria  is  concerned  with  the  ultimate  value  of  the 
training  program  to  the  organisation  (Muchinsky,  1993).  This  is  a  notoriously  difficult 
criterion  to  evaluate.  Whileno  formal  utility  analysis  was  undertaken  as  part  of  this  study,  the 
virtual  mission  preparation  provided  in  Black  Skies  08  had  obvious  practical  benefits  for  both 
organisational  and  operational  elements  with  in  Pitch  Black  08.  These  benefits  centred  around 
maxi  mi  sing  thetraining  benefits  of  an  extremely  costly  and  resource-intensive  event  such  as 
Pitch  Black  08. 

First,  the  White  Force  el  ementsresponsiblefor  developing  and  implementing  the  Pitch  Black 
08  missions  benefited  considerably  from  their  involvement  in  Black  Skies  08,  as  the  virtual 
mi  ssi  on  preparati  on  served  to  hi  ghl  i  ght  defi  ci  end  es  i  n  the  scenari  os  that  were  pi  anned  for  the 
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subsequent  live  exercise.  For  example,  it  became  apparent  during  Black  Skies  08  that  aspects 
of  the  scenari  o  were  I  i  kel  y  to  i  ncrease  the  possi  bi  I  i  ty  of  breakdow  ns  i  n  ai  rcraft  separati  on  i  n  a 
particular  region  of  airspace.  This  situation  arose  because  the  sensor  and  communications 
coverage  in  that  region  of  airspace  was  degraded  dueto  distance;  this  had  theconsequenceof 
reducing  the  effective  area  of  situational  awareness  and  influence  of  the  ABM  team.  This 
vignette  was  subsequently  amended  for  Pitch  Black  08,  reducing  the  risk  of  a  safety  incident 
duringthelive  exerci  se. 

Second,  the  ABM  team  that  attended  Black  Skies  08  benefited  from  the  virtual  mission 
preparation  i  n  that  they  were  ableto  more  effectively  utilise  thevaluabletraining  opportunity 
presented  by  Pitch  BI  ack  08.  According  to  this  team,  thetraining  benefits  of  the  first  few  days 
of  any  large-scale,  multi-national  exercise  are  usually  lost  in  the  process  of  familiarising 
themsel  ves  w  i  th  the  sheer  vol  u  me  of  Red  and  BI  u  e  ai  rcraft,  i  d  enti  fy  i  ng  the  most  effecti  ve  rol  e 
configuration  within  theteam,  and  refining  coordination  strategies,  procedures,  and  tactics.  It 
is  worth  poi  nti  ng  out  that  during  the  first  two  days  of  Pitch  Black  08,  there  were  roughly  one 
hundred  aircraft  airborne  for  more  than  ten  hours.  The  ABM  team  that  received  the  virtual 
mission  preparation  reported  that,  savefor  the  virtual  mission  preparation,  this  initial  period 
would  have  offered  very  little  training  benefit.  It  is  also  possible  that  the  enhanced 
preparedness  of  this  team  also  had  flow-on  benefits  for  the  training  experience  of  other 
participants  in  Pitch  Black  08. 


4.  Discussion 

The  principal  aim  of  this  study  was  to  investigate  the  benefits  of  using  a  synthetic 
environment  to  provide  mission  preparation  for  a  command  and  control  team  prior  to  a  live 
warfighting  exercise.  Our  specific  research  questions  werecouched  within  Kirkpatrick's(1959) 
framework  for  the  evaluation  of  training  programs:  Reaction,  Learning,  Behavioural,  and 
Results. 

In  respect  to  participants'  reactions  to  the  virtual  mission  preparation,  the  members  of  the 
ABM  team  that  participated  in  Black  Skies  08  reported  that  the  exercise  had  considerable 
combat  mission  training  value:  they  felt  it  had  a  positive  impact  on  their  team  coordination, 
their  tactical  skills,  and  their  overall  combat  mission  readiness.  Significantly,  they  believed 
that  it  provided  a  learning  experience  not  available  in  their  regular  program  of  training.  In 
fact,  the  ABM  team  reported  that  Black  Skies  08  compared  favourably  to  both  regular  training 
and  live  exercises  such  as  Pitch  Black  08  in  terms  of  the  capacity  to  provide  training 
experiences  critical  to  the  ABM  role.  The  synthetic  environment  was  viewed  as  inferior  to  the 
I  i  ve  en v i  ron  ment  p  r i  mar i  I  y  i  n  i  ts  cap aci  ty  to  p ro v i  d  e trai  n  i  n g  i  n  i  nteracti  n g  w  i  th  a  si  g n i  f i  cant 
number  of  other  command  elements  and  external  agencies. 

I  n  terms  of  the  extent  to  w  hi  ch  BI  ack  Ski  es  08  served  as  an  effecti  ve  I  earni  ng  experi  ence,  the 
ABM  team  that  participated  in  the  virtual  mission  preparation  demonstrated  a  marked 
improvement  in  teamwork  processes,  and  reported  an  increase  in  collective  self-efficacy  and 
cohesi  on,  over  the  course  of  the  exerci  se.  They  al  so  showed  a  cl  ear  i  improvement  across  most, 
but  not  all,  of  their  mission  essential  tasks.  The  tasks  that  showed  the  most  improvement 
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included  those  associated  with  the  control  of  ai rspace  and  establishing  military  liaison;  the 
tasks  that  showed  the  I  east  i  improvement  i  nd  uded  core  DCA  tasks  such  as  the  pri  oriti  sati  on  of 
threats  and  i  ntercept  geometry  (the  discrepancy  between  the  reaction-level  findings  and  these 
learning-level  findings  is  discussed  below). 

In  relation  to  Kirkpatrick'sbehavioural-level  criteria,  the  ABM  team  that  participated  in  Black 
Skies  08  performed  better  overall  than  the  control  team  during  the  subsequent  live 
warfighting  exercise.  This  performance  advantage  was  most  evident  in  terms  of  superior 
teamwork  processes,  although  the  team  that  received  the  virtual  mission  preparation  also 
performed  better  across  most  mi ssi  on  essenti  al  tasks  on  thefi  rst  day  of  Pitch  Bl ack  08.  On  the 
second  day  of  the  live  exercise,  the  gap  in  taskwork  performance  between  thetwo  teams  was 
not  as  great. 

Fi  nal  I  y,  to  the  extent  that  it  was  possi  bl  eto  eval  uate  Ki  rkpatri  ck's  (1959)  Results  cri  teri  a  i  n  the 
timeframe  of  this  study,  Black  Skies  08  also  demonstrated  a  broader  organisational  and 
operati  onal  val  ue  to  the  RA  A  F :  i  t  provi  d ed  the  W  h i  te  Force  w  i  th  the  op portu  n i  ty  to  i  d enti fy 
deficiencies  in,  and  mitigate  risks  associated  with,  the  Pitch  Black  08  mission  scenarios;  it 
allowed  the  ABM  team  to  refine  their  Pitch  Black  08  plans  and  procedures,  enabling  them  to 
more  effectively  utilise  the  valuable  training  opportunity  presented  by  a  large-scale,  live 
warfighting  exercise. 

Thesefindings  are  con  si  stent  with  previ  ous  research  into  theuse  of  synthetic  environments  to 
provide  team  and  collective  training  for  air  operations,  the  bulk  of  which  suggests  that  it 
draws  favourable  reactions  from  participants  (McIntyre  et  al.,  2002;  Schreiber  et  al.,  2006; 
Smith  et  al .,  2006)  and  has  a  positi  ve  i  mpact  on  team  processes  and  taskwork  ski  1 1  s  (M  cl  ntyre 
etal.,  2002;  Schiflett  et  al .,  2000;  but  see  Smith  &  McIntyre,  2003).  The  current  study  makesa 
novel  and  significant  contribution  to  this  research  by  also  demonstrating  that  a  command  and 
control  team  that  received  virtual  mission  preparation  for  a  live  warfighting  exercise 
performed  better  than  a  team  that  did  not  receive  this  training. 

Broadly  speaking,  these  findings  suggest  that  providing  team  training  in  a  synthetic 
environment  holds  significant  potential  for  the  RAAF.  These  technologies  and  methods 
provide  teams  of  warfighters  with  the  opportunity  to  link  together  to  engage  in  high-level 
training  and  mission  rehearsal,  and  to  experiment  with  new  tactics  and  capabilities,  more 
frequently  -  and  in  a  more  cost-effective  manner  -  than  is  possible  using  real  platforms. 
Conducting  activities  of  this  kind  on  a  more  frequent  basis  offers  the  potential  to  enhance 
operational  readiness  through  training,  and  to  accelerate  improvements  in  organisational 
processes  through  experimentation. 

4.1  Limitations  of  this  Study  and  Future  Research  and  Development 

4.1.1  Discrepancy  between  Reaction- Level  and  Learning-Level  Outcomes 

In  this  study  there  were  some  contradictory  findings  which  require  further  discussion. 
Although  thevirtual  mission  preparation  resulted  in  improved  performanceon  most  mi  ssi  on 
essential  tasks  over  the  course  of  Black  Skies  08  and  subsequently  during  Pitch  Black  08,  ithad 
I  ittle  i  mpact  on  a  subset  of  tasks  di  recti  y  concerned  with  the  conduct  of  DCA  missions;  these 
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tasks  included  the  effective  employment  of  forces  and  the  management  of  rules  of 
engagement.  This  finding  is  surprising  given  that  the  ABM  team  that  received  the  virtual 
mission  preparation  reported  that  Black  Skies  08  provided  a  more  valuable  training 
experience  for  these  core  defensive  counter  air  tasks  than  for  any  other  category  of  mission 
essential  tasks.  At  a  broader  level,  this  inconsistency  between  reaction-level  results  and 
I  earni  ng-l  evel  resul  ts  hi  ghl  i  ghts  both  the  I  i  mi  tati  ons  of  rel  yi  ng  on  reach  on-l  evel  data  to  d  raw 
cond  usions  about  the  effectiveness  of  a  trai  ni  ng  program  (T  annenbaum  &  Yukl ,  1992),  and  the 
benefits  of  evaluating  the  effectiveness  of  training  programs  at  different  levels  (Kirkpatrick, 
1959)  or  on  different  dimensions  (Kraiger,  2003). 

There  are  a  number  of  possible  explanations  as  to  why  no  benefit  was  observed  on  this 
training  objective  over  the  course  of  the  study.  First,  it  is  possible  that  the  measure  we 
empl  oyed  was  not  sensiti  veto  variabi  lity  in  the  performance  of  DC  A  tasks  duri  ng  Black  Skies 
08  and  Pitch  Black  08(i.e.,  the  measure  was  not  sufficiently  valid).  Second,  the  scenarios  in 
Black  Skies  08  and  Pitch  Black  08  may  not  have  been  sufficiently  complex  or  demanding  to 
elicit  a  superior  demonstration  of  DCA  skills  from  either  ABM  team  (although  this 
explanation  would  also  run  counter  to  reaction-level  findings).  Third,  it  is  possible  that 
Fighter  Controllers  are  so  well  practiced  in  this  aspect  of  their  role  that  the  few  days  of 
trai  ni  ng  provi  ded  i  n  Bl  ack  Ski  es  08  had  no  appreci  abl  e  i  impact  on  thei  r  performance  at  a  team 
I  evel .  Thi  s  i  nterpretati  on  i  s  consi  stent  with  previ  ous  research  showi  ng  a  I  ack  of  i  improvement 
i  n  rol  e-specifi  c  ski  1 1  s  over  the  course  of  syntheti  c  trai  ni  ng  exerci  ses  (Best  et  al .,  2007;  Smith  & 
M  cl  ntyre,  2003). 

It  would  not  bewi  se  to  cond  udefromonecase  study,  and  against  the  general  trend  of  results, 
that  these  particular  DCA  skillscan  not  be  trained  in  a  synthetic  environment.  Thisfinding 
does,  however,  serve  to  highlight  the  point  raised  by  Smith  and  McIntyre  (2003):  the  use  of 
networked  simulators  will  not,  in  and  of  itself,  guarantee  the  learning  of  critical  skills  or 
improvements  in  performance.  Further  work  is  required  to  fully  understand  the  factors  that 
determine  the  effectiveness  of  such  training. 

4.1.2  Future  Research  and  Development 

Thisstudy  has  in  fact  generated  several  avenues  for  further  research,  both  specificand  broad. 
The  findings  that  have  emerged  from  this  study  suggest  three  specific  areas  in  which  further 
w  ork  i  s  w  ar  ranted .  A  s  i  nti  mated  above,  one  of  the  pri  nci  pal  goal  s  w  i  1 1  be  to  i  d  enti  fy  the  I  ocus 
ofthediscrepancy  weobserved  b&ween  reaction-level  findings  and  learning-level  findingsin 
relation  to  core  DCA  tasks.  A  second  goal  is  to  examine  the  extent  to  which  the  performance 
benefits  associated  with  participating  in  virtual  mission  preparation  decay  overtime:  in  the 
current  study,  we  were  abl  e  to  assess  the  performance  of  A  BM  T earns  onl y  on  the  fi  rst  two 
days  of  the  Pitch  Bl  ack  08;  i  n  future  studi  es  we  hopeto  eval  uatethe  i  impact  of  vi  rtual  mi  ssi  on 
preparation  on  subsequent  performance  over  theduration  of  a  live  warfighting  exercise.  A 
third  goal  i  s  to  extend  the  general  i  sabi  lity  of  ourfindi  ngs  by  exami  ni  ng  the  i  impact  that  vi  rtual 
mission  preparation  has  on  other  command  and  control  teams,  as  well  as  warfighting  teams 
moregenerally  (such  astheaircrewJointTerminal  Attack  Controller,  and  other  elements  that 
makeup  a  close  air  support  team). 


26 


DSTO-RR-0344 


More  broadly,  thefindings  of  this  study  -  and  the  findings  of  the  body  of  work  on  which  it 
builds  -  suggests  that  there  are  many  aspects  of  team  and  collective  training  in  synthetic 
envi  ronments  whi  ch  requi  re  further  research.  These  i  ncl  ude  (1)  the  i  ntegrati  on  of  I  i  ve,  vi  rtual , 
and  constructive  systems;  (2)  the  development  of  tools  to  facilitate  collaboration  during 
distributed  mission  planning  and  debriefing;  (3)  the  development  and  evaluation  of  more 
effi  ci  ent  methods  for  the  creati  on  and  management  of  trai  ni  ng  content;  (4)  val  i  d  and  rel  i  abl  e 
measurement  of  team  and  collective  performance;  and  (5)  methods  of  providing  rapid  and 
detailed  after  action  review. 

In  order  to  more  efficiently  address  these  and  other  research  questions,  DSTO  is  proposing  to 
take  part  in  a  collaborative  program  of  research  involving  DSTL  in  the  UK,  the  Canadian 
Forces  Experimentation  Center,  the  Defence  Technology  Agency  in  New  Zealand,  and 
I  aboratori  es  withi  n  the  U  S  A  i  r  Force,  N  avy,  and  A  rmy.  Thi  s  program  of  research  wi  1 1  not  only 
allow  the  RAAF  (and  the  ADF  more  broadly)  to  leverage  the  substantial  investments  in 
syntheti  c  trai  ni  ng  capabi  I  iti  es  that  have  al  ready  been  made  by  our  pri  nci  pal  al  I  i  es,  but  i  t  w  i  1 1 
serve  to  enhance  future  interoperability  with  the  forces  of  the  participating  nations. 

4.2  Conclusion  and  Implications  for  RAAF  Synthetic  Training  Capability 

Decision  superiority  -  the  ability  to  make  and  implement  more  informed  and  more  accurate 
deci  si  ons  at  a  rate  faster  than  theenemy  -  i  s  at  the  heart  of  future  RAAF  capabi  lity  (Air  Power 
Development  Centre,  2007).  The  ability  of  the  RAAF  to  achieve  and  maintain  decision 
superiority  will  depend  on  the  professional  mastery  of  its  warfighters;  this  professional 
mastery  will  in  turn  depend  on  the  extent  to  which  they  have  the  opportunity  to  undertake 
high-level  training,  mission  rehearsal,  and  experimentation  on  a  regular  basis.  Given  its 
resource  constraints,  the  RAAF  hasidentified  virtual  and  simulated  envi  ronments  as  crucial 
to  providing  the  opportunity  for  its  warfighters  to  develop  professional  mastery.  The 
outcomes  from  this  study  aresi  gnifi  cant  in  thatthey  provide  initial  supportfor  the  previously 
untested  doctri  nethat  trai  ni  ng  i  n  a  syntheti  c  envi  ronment  can  havea  si  gnifi  cant  i  impact  on  the 
ability  of  RAAF  command  and  control  teams  to  perform  their  mission  in  large-scale,  complex, 
and  dynamic  warfighting  situations. 

O n  the  basi  s  of  the  outcomes  of  thi  s  study ,  and  the  U  S  and  U  K  research  on  w  hi  ch  thi  s  study 
builds,  we  are  in  a  position  to  make  five  key  recommendations  concerning  the  synthetic 
training  capability  of  the  RAAF.  The  recommendations  are  displayed  in  Table  2  (overleaf), 
and  discussed  further  below. 
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T able2:  R ecommendations  for  RAAF  synthetic  training  capability 


Recommendations  for  RAAF  Synthetic  Training  Capability 

1  RAAF  should  expand  its  capacity  to  provide  synthetic  team  and  collective  training  for  its 
warfighters 

2  RAAF  should  support  further  research  into  the  effective  and  efficient  use  of  synthetic  team  and 
collective  training  technology 

3  RAAF  (andtheADF  more  broadly)  should  invest  in  dedicated  infrastructure  for  synthetic  team, 
collective,  and  joint  training 

4  RAAF  should  utilise  DSTO’s  research  experience  to  guidefurther  investment  in,  and  application 
of,  tools  and  method  for  synthetic  team  and  collective  training. 

5  RAAF  should  expand  its  participation  in  coalition  synthetic  training  exercises,  and  facilitate  the 
conduct  of  collaborative  research  and  development  with  coalition  partners 


The  principal  recommendation  is  that  the  RAAF  should  expand  its  capacity  to  provide 
synthetic  team  and  collective  training  for  its  warfighters.  This  recommendation,  however, 
comes  with  two  significant  caveats  and  associated  recommendations.  Thefirst  caveat  is  that, 
as  outl  i  ned  earl  i  er  i  n  the  di  scussi  on,  further  research  i  s  requi  red  i  nto  factors  that  i  nfl  uencethe 
effecti  veness  and  effi  ci  ency  of  team  and  col  I  ecti  vetrai  ni  ng  i  n  syntheti  c  envi  ronments,  i  n  order 
to  ensure  that  the  benefit  of  such  trai  ni  ng  i  s  maxi  mi  sed.  The  second  caveat  is  that  the  uti  I  ity  of 
simulation  assets  is  constrained  by  their  method  of  employment  and  by  their  accessibility. 
Recognising  this  fundamental  constraint,  other  nations  have  developed  dedicated  mission 
trai  ni  ng  and  experi  mentati  on  faci  I  iti  es  -  such  as  the  Di  stri  buted  M  i  ssi  on  Operati  ons  Centre  i  n 
the  US,  and  the  Air  Battlespace  Training  Centre  in  the  UK  -  that  are  distinct  from  the 
i  nf  rastr  u  ctu  re  u  sed  f  or  p  u  r  p  oses  su  ch  as  p  roced  u  r  es  tr  ai  n  i  n  g .  1 1  i  s  u  n  I  i  kel  y  that  th  e  R  A  A  F '  s 
existing  simulation  assets  are  sufficient  to  realise  the  full  potential  for  team  and  collective 
training  offered  by  synthetic  envi  ronments.  Wetherefore  recommend  that  the  RAAF  (and  the 
ADF  morebroadly)  develop  dedicated  infrastructureforthis  purpose,  and  that  RAAF  utilise 
DSTO's  research  experience  to  guide  both  this  investment  and  thesubsequent  application  of 
thesyntheti  ctrai  ning  capability.  Finally,  we  recommend  thatthe  RAAF  seizetheopportunity 
to  benefit  from  the  significant  investment  that  our  principal  allies  have  already  made  in 
collectiveand  joint  training  in  synthetic  envi  ronments.  The  RAAF  can  do  so  by  increasing  its 
level  of  participation  in  coalition  synthetic  training  exercises,  and  by  supporting  and 
facilitating  the  conduct  of  collaborative  research  and  development  between  DSTO  and  the 
laboratories  of  partner  nations. 
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Appendix  A:  Descriptive  Statistics  from  Trainee 
Reactions  Questionnaire  Items 


TableAl:  Mean  Ratings  of  A  greement  with  Items  in  theFighter  Controller  Reactions  Survey 


Categories  and  Items 

Overall  Utility  and  Potential  Applications  of  Virtual  Mission  Preparation 

Th i  s  experi  ence  has  moti  vated  me  to  seek  si  m i  I  ar  trai  n  i  ng  op portu n  i  ti  es 
Participating  in  EBSwasagood  use  of  my  time 
I  would  recommend  this  experience  to  other  controllers 
Exercise  like  EBS  should  be  part  of  our  readiness  training  schedule 
I  would  liketo  see  MTDS  operational  applications  expanded 
Exercises  I  ike  EBS  should  beapartofall  future  spin-up  exercises 
MTDS  has  considerable  combat  mission  training  value 
EBS  has  limited  military  training  value 

Impact  of  Virtual  Mission  Preparation  on  Mission  Skills  and  Readiness 

This  exercise  has  positively  impacted  my  combat  mission  readiness 
MTDS  has  improved  my  combat  mission  readiness 
I  am  confident  I  will  perform  better  as  a  controller  asa  result  of  MTDS 
The  material  I  learned  in  MTDS  is  useful  for  my  future  controller  experience 
Asa  result  of  this  training,  I  have  improved  my  mission  planning  skills 
I  haveimproved  my  tactical  ski  I  Is  as  a  result  of  the  MTDS  experience 
MTDSimproved  my  understanding  of  critical  combat  skills 
I  improved  my  briefing  and  debriefing  skills 

I  feel  I  will  be  better  prepared  to  brief  and  lead  a  mission  dueto  this  exercise 

EBS  has  helped  me  improve  my  team  coordination  skills 

This  exercise  helped  me  improve  my  combat  situational  awareness 


M  SD 


6.00  0.00 

6.25  0.96 

6.75  0.50 

6.50  0.58 

4.75  1.89 

6.75  0.50 

5.75  1.50 

2.00  0.82 


6.50  0.58 

5.50  0.58 

6.25  0.96 

5.00  2.00 

3.75  1.26 

5.50  1.29 

4.75  0.96 

3.75  1.26 

6.25  0.50 

6.00  1.15 

5.50  0.58 


Specific  Training  Opportunities  Provided  by  Virtual  Mission  Preparation 

EBS  provided  an  excellent  opportunity  to  interact  with  people  systems  I  will  interact 
with  in  actual  combat 

EBS  offered  an  excellent  opportunity  to  experience  multiple  complex  presentations 

EBS  provided  valuable  experience  in  large  force  training 

Thistraining  provided  excellent  experience  in  radar  mechanics 

This  exercise  provided  excellent  practice  on  my  communication  standards 

MTDS  provides  valuable  experience  in  combat  mission  tactics 

EBS  provided  an  excel  lent  opportunity  for  me  to  practice  my  control  skills 

EBS  provided  an  excel  lent  opportunity  for  me  to  practice  my  team  coordination  skills 

EBS  provided  a  limited  opportunity  to  train  my  combat  ski  I  Is 


6.50  1.00 

5.75  1.26 

6.25  0.96 

3.25  0.96 

5.75  0.50 

5.00  1.41 

6.50  1.00 

6.50  0.58 

3.25  2.06 
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Categories  and  Items 


M  SD 


Relation  Between  Virtual  Mission  Preparation  and  Regular  Training 

I  can  routinely  get  the  type  of  experience!  had  in  MTDSatmy  homeunit  2.75 

Atmy  unit,  I  routinely  get  to  control  aircraft  against  realistic  threats  4.25 

I  can  routinely  gettraining  similar  to  that  provided  in  EBSat  homeunit  1.75 

I  routinelygettopracticetacticsinmultipleaircraftenvironmentsatmy operational  unit  4.50 
I  will  havean  opportunity  to  use  what  I've  learned  in  EBSat  my  operational  unit  6.75 

My  home  unit's  daily  OPSTEM  PO  will  help  me  maintain  theski  I  Is  I  have  learned  in  EBS  3.50 


1.50 

2.22 

0.50 

1.73 

0.50 

1.73 


Fidelity  of  the  Virtual  Mission  Preparation 

The  controller  console  was  realistic  6.75 

The  missions  were  realistic  5.25 

The  exerci  se  envi  ron  ment  was  a  real  i  sti  c  representati  on  of  combat  4.00 

MTDS  provides  realistic  training  and  experience  5.50 

MTDS  missions  and  engagements  accurately  represent  how  things  happen  in  the 
operational  world  3.75 

Generally,  the  fidelity  of  the  MTDS  environment  is  sufficient  to  achieve  the  desired 
mission/ training  objectives  5.75 

The  threats  in  this  exercise  behaved  as  they  do  in  the  operational  environment  4.75 

The  EBS  missions  I  experienced  were  unrealistic  2.75 

The  ski  I  Is  I  trained  in  EBS  are  the  same  as  those  I  am  expected  to  perform  in  the  field  6.00 


0.50 

1.50 

1.83 

1.29 

1.50 

0.50 

1.89 

0.96 

0.82 


Training  Design  and  Procedure 

The  learning  objectives  were  presented  clearly 

The  missions  helped  me  to  effectively  meet  my  learning  objectives 

In  training,  I  was  ableto  apply  lessons  learned  from  previous  missions  to  new  ones 

The  scenarios  were  well -designed 

I  felttheflow  of  EBS  was  appropriately  paced 

It  was  difficult  to  keep  up  with  the  pace  of  the  missions 

The  missions  were  challenging 

The  missions  and  engagements  were  very  challenging 


5.25  1.71 

6.25  0.96 

6.00  0.82 

5.75  0.50 

4.25  2.22 

2.00  0.82 

5.75  0.96 

5.25  1.71 


Note:  The  general  term  Mission  Training  via  Distributed  Simulation  (MTDS)  isused  in  these  items  to  refer  to  the 
kind  of  training  experience  provided  by  Black  Skies  08. 
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TableA2:  M  ean  ratings  oftheextent  to  which  it  is  possible  to  provide  critical  ABM  team  experiences 
in  thevirtual  exercise,  liveexercise,  and  home  unit  training  environments 


Air  Battle  Management  Team  Experience 

Training  Environment 

Virtual 

exercises 

Live  exercises 

Training  at 
home  unit 

M 

SD 

M 

SD 

M 

SD 

Liason/  briefer  in  a  contingency  operation  or  exercise 

1.50 

0.58 

2.75 

1.26 

2.00 

1.41 

Various  ROE 

3.50 

1.00 

3.25 

0.96 

2.75 

1.50 

Operations  near  neutral  airspace 

3.00 

0.82 

2.75 

0.96 

1.75 

0.96 

A  i  rspace  restri  cti  ons 

3.75 

0.50 

3.50 

0.58 

2.75 

0.96 

Joint  operations-  planning 

2.25 

1.26 

3.00 

1.15 

1.25 

0.50 

Joint  operations  -  integrating,  communicating,  identifying 

1.75 

1.71 

2.75 

0.96 

1.25 

0.50 

Combined  operations-  planning 

Combined  operations-  integrating,  communicating, 

1.75 

1.71 

3.25 

0.96 

1.50 

0.58 

identifying 

1.75 

1.71 

3.00 

1.15 

1.50 

0.58 

Face  to  face  mission  planning 

1.25 

0.96 

2.25 

2.06 

2.50 

1.29 

Mission  documentation/  reconstruction/  intel  debrief 

2.00 

1.63 

2.25 

1.50 

1.75 

0.50 

Working  in  a  mission  planning  team 

2.00 

1.83 

2.25 

2.06 

1.00 

0.82 

External  coordination  with  ATC  or  airspace  owner 

1.00 

0.82 

3.25 

0.96 

3.50 

0.58 

Working  with  electronic  warfare  assets  on  VPN 

1.25 

1.26 

2.75 

1.50 

0.50 

0.58 

Working  with  electronic  attack  assets 

1.75 

1.50 

2.50 

1.73 

0.25 

0.50 

Coordinate  through  linkswith  other  interface  units 

1.75 

1.71 

2.50 

1.73 

0.50 

1.00 

Force  marshal  1  i  ng  /  Battle  management 

3.50 

0.58 

3.75 

0.50 

2.75 

0.96 

Working  with  various  ground  parties 

2.00 

1.41 

2.50 

1.73 

0.50 

0.58 

Coordination  with  other  C2ISR  players 

2.00 

1.41 

2.75 

1.50 

1.00 

0.82 

Working  in  a  radar/  IFF  jamming  environment 

2.50 

1.91 

2.25 

1.26 

0.75 

0.96 

Working  in  a  communications  jamming  environment 

2.50 

1.91 

2.50 

1.29 

1.00 

1.15 

Working  in  a  dense  emitter  environment 

2.75 

1.89 

2.50 

1.00 

1.25 

0.50 

Detecting  different  types  of  targets 

3.50 

0.58 

2.50 

1.29 

2.00 

1.63 

Various  geography 

3.25 

0.96 

2.75 

0.96 

1.75 

0.96 

Various  environments 

2.25 

2.06 

2.75 

0.96 

2.50 

1.29 

Identifying  various  types  of  entities 

2.00 

1.83 

2.25 

0.96 

1.25 

1.50 

C 1  ose  control  stem  V 1 D 

3.25 

0.96 

1.25 

1.26 

1.75 

0.96 

Detecting,  1 D,  comm.,  tracking,  prosecuting  a  FI  FF 

3.50 

0.58 

3.50 

1.00 

1.75 

0.50 

Apply  ID  matrix 

3.50 

1.00 

3.50 

1.00 

3.50 

0.58 

Strangers/  unknowns  in  the  airspace  including  normal 

ATC  traffic 

3.50 

0.58 

1.25 

0.50 

1.75 

0.96 

Working  with  a  degraded  picture 

3.75 

0.50 

2.50 

1.00 

2.00 

1.41 

Describing  non  standard  group  formations 

3.25 

1.50 

3.75 

0.50 

2.75 

0.50 

Red  air  tracking  6  or  more  groups 

3.75 

0.50 

2.75 

0.96 

0.75 

0.50 

Blue  air  tracking  6  or  more  groups 

3.75 

0.50 

3.00 

1.15 

1.00 

0.82 

Update  track  info 

2.75 

0.96 

2.75 

0.96 

2.50 

1.00 

A  variety  of  enemy  threats  and  tactics 

3.75 

0.50 

2.75 

0.96 

1.25 

0.50 

A  variety  of  friendly  capabilities  and  tactics 

3.00 

1.41 

3.00 

1.41 

1.25 

0.50 
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Air  Battle  Management  Team  Experience  Training  Environment 


Tracking  radar  only  and  /  or  mode 4 only 
Degraded  comm. 

Dense  internal  and  external  comm  environment 
Dense  and  /  or  complex  link  environment 
Working  with  HAVE  QUICK  comm. 

Working  with  secure  comm. 

Maintain  safe  control  6  or  more  groups 
Air  refueling 
WD  directed  air  refueling 
Tanker  management 
AOC  coordination 

Dynamic  ground  and  air  enemy  order  of  battle 
Working  as  check-in  controller 
Controlling  a  large  strike  package 
Rapidly  changing  targets  and  priorities 
Position  E-3  orbit 

Emergency  procedures  on  board  and  off  board 
Keeping  track  of  flight  and  time  on  target  deconfliction 
HVAA  protection 

Working  with  CSAR  assets/  working  with  Hook  112 
Multiplecontingency/  fragged  missions 
Working  a  split  frequency  war 
Leaker  scenario 

Return  to  force  procedures/  lame  duck  procedures/  safe 
passage 

Station  handover 

Troubleshooting  primary  and  secondary  sensors 
Hands-on  platform  familiarisation 
Procedural  control 
Dynamic  ACMs 

Joint  fire _ 


Virtual 

exercises 

Live  exercises 

Training  at 
home  unit 

M 

SD 

M 

SD 

M 

SD 

3.25 

0.50 

1.75 

0.96 

1.00 

0.82 

2.75 

1.50 

2.50 

0.58 

1.25 

1.26 

2.75 

1.50 

2.75 

0.50 

1.50 

0.58 

2.25 

2.06 

1.00 

1.15 

0.25 

0.50 

2.00 

1.63 

2.50 

1.29 

1.75 

1.26 

2.00 

1.63 

2.25 

0.96 

1.25 

0.50 

3.50 

0.58 

2.25 

0.50 

1.00 

0.82 

3.50 

0.58 

3.00 

0.82 

0.75 

0.50 

2.25 

1.71 

2.25 

0.50 

0.75 

0.96 

3.25 

0.96 

2.50 

0.58 

0.75 

0.96 

2.75 

1.50 

3.00 

0.82 

0.50 

0.58 

3.25 

0.96 

3.00 

0.82 

1.00 

0.00 

3.25 

0.50 

3.75 

0.50 

1.50 

1.00 

3.75 

0.50 

2.75 

1.50 

0.25 

0.50 

3.25 

0.50 

2.75 

0.96 

1.25 

0.96 

3.25 

0.96 

2.25 

0.50 

0.25 

0.50 

3.50 

0.58 

1.75 

1.71 

2.00 

1.83 

3.00 

0.82 

3.50 

0.58 

1.75 

0.96 

3.50 

0.58 

3.00 

0.82 

1.00 

0.00 

2.00 

0.82 

2.50 

1.29 

0.50 

0.58 

3.25 

0.96 

2.25 

0.50 

0.25 

0.50 

3.50 

0.58 

3.00 

0.82 

0.75 

0.50 

3.50 

0.58 

3.50 

0.58 

1.75 

0.96 

3.25 

0.96 

2.50 

1.29 

0.25 

0.50 

3.00 

1.15 

2.50 

1.00 

1.75 

1.71 

3.25 

0.96 

3.00 

1.41 

2.75 

1.50 

1.50 

1.73 

2.25 

0.96 

1.25 

1.50 

3.25 

0.50 

2.75 

0.96 

0.75 

0.50 

3.75 

0.50 

3.25 

0.50 

1.00 

0.82 

3.50 

0.58 

3.25 

0.50 

1.00 

0.82 
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